Sample for client RESTful API, pandas, tqdm, argparse

    Here is a good example for practicing:

    DataQuest: Tutorial: Getting Music Data with the Last.fm API using Python

    Why I didn't mention it as "good tutorial" because I encounter several obstacles while working on this example (then it became a project..) Probably 

• related tools' reversion is the main issue. IT is a fast changing world. Many tutorials I found online are already invalid. 
• the author was using both programming and terminal. I was mainly on programming because data processing/conversion in advance;
• Skipping complex  work, ex. temporary storage. 

    At beginning I expected to complete this exercise in one file. But at the end, to deal with multiple obstacles, It grew to mobilized functions (options/database/REST loading/parsing) , many steps ( at least 7, not including removed tests) and many tests..... 

    In order to deal with multiple REST querying and its agreement of usage, 

I employed SQLite3. At first, I was using saving-files to keep records instead of harassing server while debugging. I thought of saving multiple files during multiple REST queries and save all into one file at the end. I assume saving file is convenient to verify content, since I don't have GUI viewers for SQLite file format. But, considering the possibility that other programmers/learners may want to test on my project and querying REST thousands of times, creating thousands of files and deleting them in a sudden seems suspicious... I can't guarantee REST query will always succeed, nor my program was robust,  so saving-data is on progress, instead of at the end. Therefor I developed tier 2 for database storage. And later on, some options only provide database storage. 

    But as expected, it is inconvenient to view the raw data to decipher its scheme. Many database related testing functions are only for viewing purpose. And then adding action_id to distinguish what is this data for, some are from REST query, some are rebuilt JSON after processing. 

    My project on GitHub: https://github.com/atfuture7/learning/tree/master/python/09_restfm

    Here are some modules the author mentioned or I found during developing. 

Module: argparse

parser = argparse.ArgumentParser()

parser.add_argument(...)    

args = parser.parse_args()

Python.Docs>>argparse — Parser for command-line options, arguments and sub-commands

Docs in python is detailed, and this module can also take function to check valid values. but how many keyword it adapt.....? A ha~ I found one occurrence of string type on this page... And required tag for the value of omittable argument is so far away.... 

Here is a simple version if you don't want to be a argparse expert yet. 

gitconnected>> The Easy Guide to Python Command Line Arguments

   This module make expanding arguments much easier. 

Module: pandas 

API Reference for pandas 

    This is the module took me most of time while developing, because, on tutorial or most online examples, developers are using them on terminal. Its functionality on terminal is very muck alike programming. After all other bugs fixed and still nothing showing out, I finally discovered that something showing on terminal, didn't return by code..... And something absolutely different on output wasn't changed on object... And even when the view are the same, they are under different axis-definition at import.... And this make one column merge failed..... 

    This is my 2nd time visited module pandas. At first time, I don't know what to do for practicing. This tutorial proposed good samples. 

    Yes, pandas is powerful

Module: tqdm

Documentation on tqdm Home

    This is a simple and amusing progressing bar. I see it often when updating developing environment. It also forced me to understand pandas.DataFrame.apply(). What would be pushed in as parameters and how I can add extras,  

(But it seem that pandas is going to remove panel sharing.... Which tqdm depends on.  Let's see how would it go.)