Complete the steps described in the rest of this page to create a simple Python command-line application that makes requests to the Gmail API.
2018-11-15
What problem does pandas solve?
Python has long been great for data munging and preparation, but less so for data analysis and modeling. pandas helps fill this gap, enabling you to carry out your entire data analysis workflow in Python without having to switch to a more domain specific language like R.
From <https://pandas.pydata.org/>
NumPy
NumPy is the fundamental package for scientific computing with Python. It contains among other things:
• a powerful N-dimensional array object
• sophisticated (broadcasting) functions
• tools for integrating C/C++ and Fortran code
• useful linear algebra, Fourier transform, and random number capabilities
From <http://www.numpy.org/>
Array Broadcasting
Broadcasting is the name given to the method that NumPy uses to allow array arithmetic between arrays with a different shape or size.
From <https://machinelearningmastery.com/broadcasting-with-numpy-arrays/>
scikit-learn
Machine Learning in Python
• Simple and efficient tools for data mining and data analysis
• Accessible to everybody, and reusable in various contexts
• Built on NumPy, SciPy, and matplotlib
• Open source, commercially usable - BSD license
From <https://scikit-learn.org/stable/>
Welcome to PyBrain
PyBrain is a modular Machine Learning Library for Python. Its goal is to offer flexible, easy-to-use yet still powerful algorithms for Machine Learning Tasks and a variety of predefined environments to test and compare your algorithms.
From <http://pybrain.org/>
2018-11-06
python read fails on special characters
with io.open(fileToSearch,'r',encoding='utf-8') as file:
From <https://stackoverflow.com/questions/47635759/how-to-read-a-text-file-with-special-characters-in-python>
An unrelated hint: have a look at the built-in function enumerate, which frees you from taking care of incrementing counter: You simply write for counter, line in enumerate(file):
From <https://stackoverflow.com/questions/47635759/how-to-read-a-text-file-with-special-characters-in-python>
2018-10-31
A number of IDEs support menu options to set the execution environment for programs under development and testing. In particular, it would be nice if IDLE let the user set command line arguments to be passed into sys.argv when running a script by pressing F5.
Here are some existing implementations for reference:
* Wing-IDE: https://wingware.com/doc/intro/tutorial-debugging-launch
* Visual Studio: https://www.youtube.com/watch?v=IgbQCRHKV-Y
* PyCharm: https://www.jetbrains.com/pycharm/help/run-debug-configuration-python.html
This feature will help users interactively develop and test command-line tools while retaining all the nice features of the IDE. I would personally find it useful when teaching students about how sys.argv works.
From <https://bugs.python.org/issue5680>
Pending application of a patch, the following will work to only add args to sys.argv when running from an Idle editor.
import sys
# …
if name == 'main':
if 'idlelib.PyShell' in sys.modules:
sys.argv.extend(('a', '-2')) # add your argments here.
print(sys.argv) # in use, parse sys.argv after extending it
# ['C:\\Programs\\python34\\tem.py', 'a', '-2']
From <https://bugs.python.org/issue5680>
try:
__file__
except:
sys.argv = [sys.argv[0], 'argument1', 'argument2', 'argument2']
From <https://stackoverflow.com/questions/2148994/when-running-a-python-script-in-idle-is-there-a-way-to-pass-in-command-line-arg>
2018-10-31
Auto detect IDLE and prompt for command-line argument values
#! /usr/bin/env python3
import sys
def ok(x=None):
sys.argv.extend(e.get().split())
root.destroy()
if 'idlelib.rpc' in sys.modules:
import tkinter as tk
root = tk.Tk()
tk.Label(root, text="Command-line Arguments:").pack()
e = tk.Entry(root)
e.pack(padx=5)
tk.Button(root, text=“OK”, command=ok,
default=tk.ACTIVE).pack(pady=5)
root.bind(“<Return>”, ok)
root.bind("<Escape>", lambda x: root.destroy())
e.focus()
From <https://stackoverflow.com/questions/2148994/when-running-a-python-script-in-idle-is-there-a-way-to-pass-in-command-line-arg/44687632#44687632>
2018-10-30
Python Script
Here's a Python script that will also print the files deleted
import os
for line in open(“./data/deleted.files”):
if line.isspace() or line[0] == '#':
continue
line = line.rstrip(os.linesep)
try:
if os.path.exists(line):
print('File removed => ' + line)
os.remove(line)
except OSError:
pass
Here's an alternative Python script that is case sensitive and will also delete directories included in the list
import os
import shutil
def exists_casesensitive(path):
if not os.path.exists(path):
return False
directory, filename = os.path.split(path)
return filename in os.listdir(directory)
with open(“./data/deleted.files”) as file:
for line in file:
line = line.strip()
if line and not line.startswith('#'):
path = line.rstrip(os.linesep)
if exists_casesensitive(path):
if os.path.isdir(path):
shutil.rmtree(path)
print('Directory removed => ' + path)
else:
os.remove(path)
print('File removed => ' + path)
else:
#print('File not found => ' + path)
pass
From <https://www.dokuwiki.org/install:unused_files>
2018-10-06
GLOB
def delete_previous_checkpoints(self, num_previous=5):
"""
Deletes all previous checkpoints that are <num_previous> before the present checkpoint.
This is done to prevent blowing out of memory due to too many checkpoints
:param num_previous:
:return:
"""
self.present_checkpoints = glob.glob(self.get_checkpoint_location() + '/*.ckpt')
if len(self.present_checkpoints) > num_previous:
present_ids = [self.__get_id(ckpt) for ckpt in self.present_checkpoints]
present_ids.sort()
ids_2_delete = present_ids[0:len(present_ids) - num_previous]
for ckpt_id in ids_2_delete:
ckpt_file_nm = self.get_checkpoint_location() + '/model_' + str(ckpt_id) + '.ckpt'
os.remove(ckpt_file_nm)
From <https://www.programcreek.com/python/example/92/glob.glob>
2018-10-06
If you're doing anything more complicated than a script that accepts a few required positional arguments, you'll want to use a parser. Depending on your python version, there are 3 available in the python standard library (getopt, optparse and argparse) and argparse is by far the best.
From <https://stackoverflow.com/questions/35365344/python-sys-argv-and-argparse>
Argparse Tutorial
author: Tshepang Lekhonkhobe
This tutorial is intended to be a gentle introduction to argparse, the recommended command-line parsing module in the Python standard library.
From <https://docs.python.org/3.7/howto/argparse.html>
*args and kwargs in Python
*args
The special syntax *args in function definitions in python is used to pass a variable number of arguments to a function. It is used to pass a non-keyworded, variable-length argument list.
From <https://www.geeksforgeeks.org/args-kwargs-python/>
2018-10-03
In Python 3.5 and newer use the new recursive / functionality:
configfiles = glob.glob('C:/Users/sam/Desktop/file1//*.txt', recursive=True)
When recursive is set, followed by a path separator matches 0 or more subdirectories.
From <https://stackoverflow.com/questions/14798220/how-can-i-search-sub-folders-using-glob-glob-module-in-python>
for i in d.iglob('/*'):
The “” pattern means “this directory and all subdirectories, recursively”. In other words, it enables recursive globbing:
From <https://docs.python.org/3/library/pathlib.html>
errno.ENOTEMPTY
Directory not empty
From <https://docs.python.org/2/library/errno.html>
errno.EACCES¶
Permission denied
From <https://docs.python.org/2/library/errno.html>
except OSError as e:
if e.errno not in _IGNORED_ERROS:
raise
return False
From <https://github.com/python/cpython/blob/3.7/Lib/pathlib.py>
except OSError as e:
if e.errno != EINVAL and strict:
raise
From <https://github.com/python/cpython/blob/3.7/Lib/pathlib.py>
2018-10-01
import os, sys
from stat import *
def walktree(top, callback):
'''recursively descend the directory tree rooted at top,
calling the callback function for each regular file'''
for f in os.listdir(top):
pathname = os.path.join(top, f)
mode = os.stat(pathname).st_mode
if S_ISDIR(mode):
# It's a directory, recurse into it
walktree(pathname, callback)
elif S_ISREG(mode):
# It's a file, call the callback function
callback(pathname)
else:
# Unknown file type, print a message
print('Skipping %s' % pathname)
def visitfile(file):
print('visiting', file)
if name == 'main':
walktree(sys.argv[1], visitfile)
From <https://docs.python.org/3/library/stat.html>
2018-09-29
Dropbox in python
from pathlib import Path
import arrow
filesPath = r“C:\scratch\removeThem”
criticalTime = arrow.now().shift(hours=+5).shift(days=-7)
for item in Path(filesPath).glob('*'):
if item.is_file():
print (str(item.absolute()))
itemTime = arrow.get(item.stat().st_mtime)
if itemTime < criticalTime:
#remove it
pass
From <https://stackoverflow.com/questions/12485666/python-deleting-all-files-in-a-folder-older-than-x-days>
In IDLE, go to Options → Configure IDLE → Keys and there select history-next and then history-previous to change the keys.
Then click on Get New Keys for Selection and you are ready to choose whatever key combination you want.
From <https://stackoverflow.com/questions/4289937/how-to-repeat-last-command-in-python-interpreter-shell>
2018-09-04
CSV Toolkit Overview
NOTE: THIS PROJECT HAS SINCE BEEN FORKED TO THE INTERNAL PROMETHEUS RESEACH, LLC TOOL PROPS.CSVTOOLKIT
CSV Toolkit is a Python package that provides validation tooling and processing of CSV files. The validation tooling is based on the fantastic package Vladiate. The interface and extension mechanisms are similarly implemented as the rex.core extension mechanisms.
From <https://pypi.org/project/csv.toolkit/>
What is Bonobo?
Bonobo is a lightweight Extract-Transform-Load (ETL) framework for Python 3.5+.
It provides tools for building data transformation pipelines, using plain python primitives, and executing them in parallel.
Bonobo is the swiss army knife for everyday's data.
From <https://www.bonobo-project.org/>
csvkit is a suite of command-line tools for converting to and working with CSV, the king of tabular file formats.
From <https://csvkit.readthedocs.io/en/1.0.3/>
Awesome Python
A curated list of awesome Python frameworks, libraries, software and resources.
Inspired by awesome-php.
• Awesome Python
○ Admin Panels
○ Algorithms and Design Patterns
○ Anti-spam
○ Asset Management
○ Audio
○ Authentication
○ Build Tools
○ AND MANY MORE
From <https://github.com/vinta/awesome-python>
2018-09-04
Python data visualization: Comparing 7 tools
The Python scientific stack is fairly mature, and there are libraries for a variety of use cases, including machine learning, and data analysis. Data visualization is an important part of being able to explore data and communicate results, but has lagged a bit behind other tools such as R in the past.
From <https://www.dataquest.io/blog/python-data-visualization-libraries/>
2018-09-04
Best way to sort txt file using csv tools in python
From <https://stackoverflow.com/questions/45221637/best-way-to-sort-txt-file-using-csv-tools-in-python>
import csv
import operator
#==========Search by ID number. Return Just the Name Fields for the Student
with open(“studentinfo.txt”,“r”) as f:
studentfileReader=csv.reader(f)
id=input("Enter Id:")
for row in studentfileReader:
for field in row:
if field==id:
currentindex=row.index(id)
print(row[currentindex+1]+" "+row[currentindex+2])
#=========Sort by Last Name
with open(“studentinfo.txt”,“r”) as f:
studentfileReader=csv.reader(f)
sortedlist=sorted(f,key=operator.itemgetter(0),reverse=True)
print(sortedlist)
From <https://stackoverflow.com/questions/45221637/best-way-to-sort-txt-file-using-csv-tools-in-python>
2018-08-10
The sys.path list contains the list of directories which will be searched for modules at runtime:
python -v
['', '/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', … ]
From <https://stackoverflow.com/questions/269795/how-do-i-find-the-location-of-python-module-sources>
For speedtest - /usr/local/lib