Python
Howto copy a Python3 virtual environment to another location
0If you want to copy a python3 virtual environment to another location do the following:
sudo apt install -y python3-pip build-essential \
libssl-dev libffi-dev python3-dev python3-venv
mkdir Python-Envs
cd Python-Envs
python3.6 -m venv env
Make a entry point for the various projects
cd Python-Envs
mkdir src
To activate:
cd Python-Envs
source env/bin/activate
cd src[My project]
Do your stuff
to deactivate:
deactivate
In case you want to move a virt to another machine:
In the source env do:
pip3 freeze > requirements.txt
Copy that requirements.txt file to the other virt
in the other virt do:
pip3 install -r requirements.txt
Using Python virtual environment
2Nothing special just a reminder for myself.
sudo apt-get install python-virtualenv
mkdir python-virt-env
cd python-virt-env/
~/python-virt-env$ virtualenv venv
~/python-virt-env$ source venv/bin/activate
Now your ready to go
Stop it:
~/python-virt-env$ deactivate
Get sqlautocode to work with slqalchemy >= 0.8
1As the tool sqlautocode doesn’t work with sqlaclhemy >= 0.8 I’ve patched it to get to work.
(of course I’ve send the patch to the maintainers but I fear the project isn’t very alive)
Here’s the patch:
diff -u sqlautocode/ my_sqlautocode/ diff -u sqlautocode/config.py my_sqlautocode/config.py --- sqlautocode/config.py 2014-06-06 10:29:55.000000000 +0200 +++ my_sqlautocode/config.py 2014-06-06 10:22:56.000000000 +0200 @@ -140,7 +140,7 @@ engine = sqlalchemy.create_engine(url) test = engine.connect() test.close() - except sqlalchemy.exceptions.SQLAlchemyError, ex: + except sqlalchemy.exc.SQLAlchemyError, ex: parser.error('Could not connect to "%s": %s' % (url, ex))
diff -u sqlautocode/declarative.py my_sqlautocode/declarative.py --- sqlautocode/declarative.py 2014-06-06 10:29:55.000000000 +0200 +++ my_sqlautocode/declarative.py 2014-06-06 10:08:09.000000000 +0200 @@ -14,8 +14,12 @@ from sqlalchemy.ext.declarative import _deferred_relationship except ImportError: #SA 0.5 support - from sqlalchemy.ext.declarative import _deferred_relation as _deferred_relationship - + try: + from sqlalchemy.ext.declarative import _deferred_relation as _deferred_relationship + except ImportError: + #SA 0.8 support + from sqlalchemy.ext.declarative.clsregistry import _deferred_relationship +
Syntax highlighting for Kivy kv language in pycharm
10The nice folks on the Kivy ML have provided a kv language highlighting for Pycharm 🙂
For easy retrieval I just post the original announcement together with the settings file:
For anyone using PyCharm, please find attached a file type extension that give you full syntax highlighting for KV files. Courtesy of Xuton…;-)
To install:
On Pycharm’s main menu, click File -> Manage IDE Setting -> Import settings
Select this file, then it will come with a dialog with filetypes ticked. Just click ok or whatever and then its there.
You can get the file here:
https://github.com/Zen-CODE/kivybits/blob/master/IDE/PyCharm_kv_completion.jar
Python xmlrpcserver with HTTPS support and basic authentication
6For a project I’m working on at Formatics.nl I needed a Python based multithreaded xmlrpcserver with HTTPS support, HTTP basic authentication and capable of running on any port. After looking into solutions which required some sort of server for the HTTPS and basic autentication parts I decided to go for a better solution to try to add the HTTPS support and basic authentication to the Python xmlrpsserver itself. That way we would have a simple and fast solution which could run from any box on any port. Looking around on the net I found various solutions for every requirement but nothing which combined it all into one solution. So after combining various bits and pieces I came up with a Python xmlrpcserver with HTTPS and basic authentication support.
As the code used came from various free-software projects and locations I decided to share the server with the world so that it might benefit others.
For the HTTPS you need OpenSSL certificates, use your existing PEM formatted one or just create one yourself.
$ openssl genrsa -out privkey.pem 2048 $ openssl req -new -key privkey.pem -out cert.csr $ openssl req -new -x509 -key privkey.pem -out cacert.pem -days 1095
And here’s the server
from SocketServer import ThreadingMixIn, BaseServer from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler, SimpleXMLRPCDispatcher import socket from OpenSSL import SSL from base64 import b64decode from threading import Thread, Condition from thread import start_new_thread import traceback import time # static stuff DEFAULTKEYFILE = 'privkey.pem' # Replace with your PEM formatted key file DEFAULTCERTFILE = 'cacert.pem' # Replace with your PEM formatted certificate file class Services: def give_me_time(self): return time.asctime() class CustomThreadingMixIn: """Mix-in class to handle each request in a new thread.""" # Decides how threads will act upon termination of the main process daemon_threads = True def process_request_thread(self, request, client_address): """Same as in BaseServer but as a thread. In addition, exception handling is done here. """ try: self.finish_request(request, client_address) self.close_request(request) except (socket.error, SSL.SysCallError), why: print 'socket.error finishing request from "%s"; Error: %s' % (client_address, str(why)) self.close_request(request) except: self.handle_error(request, client_address) self.close_request(request) def process_request(self, request, client_address): """Start a new thread to process the request.""" t = Thread(target=self.process_request_thread, args=(request, client_address)) if self.daemon_threads: t.setDaemon(1) t.start() class MyXMLRPCServer(CustomThreadingMixIn, SimpleXMLRPCServer): def __init__(self, ip, port, keyFile=DEFAULTKEYFILE, certFile=DEFAULTCERTFILE, logRequests=True): self.logRequests = logRequests class VerifyingRequestHandler(SimpleXMLRPCRequestHandler): def setup(myself): myself.connection = myself.request myself.rfile = socket._fileobject(myself.request, "rb", myself.rbufsize) myself.wfile = socket._fileobject(myself.request, "wb", myself.wbufsize) def address_string(myself): "getting 'FQDN' from host seems to stall on some ip addresses, so... just (quickly!) return raw host address" host, port = myself.client_address #return socket.getfqdn(host) return host def do_POST(myself): """Handles the HTTPS POST request. It was copied out from SimpleXMLRPCServer.py and modified to shutdown the socket cleanly. """ try: # get arguments data = myself.rfile.read(int(myself.headers["content-length"])) # In previous versions of SimpleXMLRPCServer, _dispatch # could be overridden in this class, instead of in # SimpleXMLRPCDispatcher. To maintain backwards compatibility, # check to see if a subclass implements _dispatch and dispatch # using that method if present. response = myself.server._marshaled_dispatch(data, getattr(myself, '_dispatch', None)) except Exception, info: # This should only happen if the module is buggy print "ERROR do_POST: ", info print "Traceback follows:", traceback.print_exc() # internal error, report as HTTP server error myself.send_response(500) myself.end_headers() else: # got a valid XML RPC response myself.send_response(200) myself.send_header("Content-type", "text/xml") myself.send_header("Content-length", str(len(response))) myself.end_headers() myself.wfile.write(response) # shut down the connection myself.wfile.flush() myself.connection.shutdown() # Modified here! def do_GET(myself): """Handles the HTTP GET request. Interpret all HTTP GET requests as requests for server documentation. """ # Check that the path is legal if not myself.is_rpc_path_valid(): myself.report_404() return response = myself.server.generate_html_documentation() myself.send_response(200) myself.send_header("Content-type", "text/html") myself.send_header("Content-length", str(len(response))) myself.end_headers() myself.wfile.write(response) # shut down the connection myself.wfile.flush() myself.connection.shutdown() # Modified here! def report_404(myself): # Report a 404 error myself.send_response(404) response = 'No such page' myself.send_header("Content-type", "text/plain") myself.send_header("Content-length", str(len(response))) myself.end_headers() myself.wfile.write(response) # shut down the connection myself.wfile.flush() myself.connection.shutdown() # Modified here! def parse_request(myself): if SimpleXMLRPCRequestHandler.parse_request(myself): basic, foo, encoded = myself.headers.get('Authorization').partition(' ') username, foo, password = b64decode(encoded).partition(':') #print username, foo, password if username == 'admin': return True else: myself.send_error(401, 'Authentication failed') return False SimpleXMLRPCDispatcher.__init__(self, False, None) BaseServer.__init__(self, (ip, port), VerifyingRequestHandler) # SSL socket stuff ctx = SSL.Context(SSL.SSLv23_METHOD) ctx.use_privatekey_file(keyFile) ctx.use_certificate_file(certFile) self.socket = SSL.Connection(ctx, socket.socket(self.address_family, self.socket_type)) self.server_bind() self.server_activate() self.funcs = {} self.register_introspection_functions() self.register_instance(Services()) # requests count and condition, to allow for keyboard quit via CTL-C self.requests = 0 self.rCondition = Condition() def startup(self): #run until quit signaled from keyboard print 'server starting; hit CTRL-C to quit...' while True: try: self.rCondition.acquire() start_new_thread(self.handle_request, ()) # we do this async, because handle_request blocks! while not self.requests: self.rCondition.wait(timeout=3.0) if self.requests: self.requests -= 1 self.rCondition.release() except KeyboardInterrupt: print "quit signaled, i'm done." return def get_request(self): request, client_address = self.socket.accept() self.rCondition.acquire() self.requests += 1 self.rCondition.notifyAll() self.rCondition.release() return (request, client_address) def listMethods(self): """return list of method names (strings)""" methodNames = self.funcs.keys() methodNames.sort() return methodNames def methodHelp(self, methodName): """method help""" if methodName in self.funcs: return self.funcs[methodName].__doc__ else: raise Exception('method "%s" is not supported' % methodName) if __name__ == '__main__': import sys if len(sys.argv) == 3: host = sys.argv[1] port = int(sys.argv[2]) else: host = 'localhost' port = 8111 server = MyXMLRPCServer(host, port, logRequests=True) # Run the server's main loop sa = server.socket.getsockname() print "Serving HTTPS on", sa[0], "port", sa[1] server.startup()
For a client use someting like this
from xmlrpclib import ServerProxy user = 'stas' pas = 'stas' p = ServerProxy('https://%s:%s@localhost:8111' % (user, pas)) print p.give_me_time()
Pycharm exclude files from beeing synchronized
0When running on a slower system or laptop synchronizing can take some time when you have a lot of binary files. The solution is to exclude them.
Settings -> Deployment -> Exclude items by name:
For example:
.svn;.cvs;.idea;.DS_Store;.git;.hg;*/DbaseAssets/Images/*;*/DbaseAssets/Sounds/*;*/builder/*/App
Building a kivy app for Android issues
0When making a kivy app for Android I came across some issues and while I informed the kivy devs I just put them here also just in case I need them later 🙂
Just follow the manual for packaging for Android.
First of all when you install the dependencies you must use python-pip iso pip.
When building the python binary I had problems with the httplib module which did have https support.
This what adb logcat told me:
I/python (19347): your version of httplib doesn’t support HTTPS
As it turned out the order in which you build the various modules matter.
So when building the Python binary you must use the order.
Put C lib first, then python lib, then kivy.
eg ./distribute.sh -m “sqlite3 libxml2 pyopenssl pil kivy”
Now there’s HTTPS support.
When you still haven’t HTTPS support try to clean the python4android git tree.
git clean -dxf
More android specific stuff I encountered:
To write to the filesystem use CWD/.kivy/….
sys.platform is on android ‘linux3’ (GNU/Linux is ‘linux2’)
Excluding files in git and egit on eclipse
0This is not rocket science but more a reminder for myself 🙂
When working with eclipse and the egit plugin I noticed that a ‘git refresh’ would take minutes in certain projects. The particular project has a few directories containing 10000+ .ogg and .png files which are not included in the git repo. I have git exclude files and global .gitignore files in place but egit seems to ignore them.
So I added the file patterns to the egit ‘Ignored Recources’.
(For example I want to exclude everything in my */lib/Sounds)
Open Window -> Preferences -> Team -> EGit -> Add pattern
The pattern to add: */lib/Sounds/* (duhhh)
Push ‘apply’ and presto, the refresh just takes seconds 🙂
Just for completeness I include my .gitignore_global, the relevant parts and the projects exclude file.
To have the git global ignoring stuff do: git config --global core.excludesfile ~/.gitignore_global
.gitignore_global:
# Custom stuff
*.project
*.pydevproject
*.settings
*.pyc
….
….
The .git/info/exclude file:
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~
builder/BtpOS/*/Data/
lib/CPData/DbaseAssets/Images/
lib/CPData/DbaseAssets/Sounds/
youtube_pl2vids.py
0Python script to download youtube vids
[sourcecode language=”python”]
import sys, os, random, time
if len(sys.argv) < 3:
print "Usage: youtube_pl2vids [playlist.html] [destination directory]"
sys.exit(1)
source = sys.argv[1]
dest = sys.argv[2]
urllines = []
f = open(os.path.join(dest, ‘youtube.dl’), ‘w’)
for line in open(source, ‘r’).readlines():
if ‘<a href="/watch?v=’ in line and ‘class="tile-link-block video-tile"’ in line:
url = "http://www.youtube.com" + line.split(‘&’)[0].split(‘<a href="’)[1]
f.write(url + ‘n’)
urllines.append(url)
f.close()
print urllines
# we assume youtube-dl is installed
for line in urllines:
os.system(‘youtube-dl -t –max-quality 22 %s’ % line)
s = random.randint(1, 6)
print "sleeping %s secs" % s
time.sleep(s)
[/sourcecode]
Recent Comments