Python

Using Python virtual environment

2

Nothing special just a reminder for myself.

sudo apt-get install python-virtualenv

mkdir python-virt-env

cd python-virt-env/

~/python-virt-env$ย  virtualenv venv

~/python-virt-env$ source venv/bin/activate

Now your ready to go

Stop it:

~/python-virt-env$ deactivate

Get sqlautocode to work with slqalchemy >= 0.8

1

As the tool sqlautocode doesn’t work with sqlaclhemy >= 0.8 I’ve patched it to get to work.

(of course I’ve send the patch to the maintainers but I fear the project isn’t very alive)

Here’s the patch:

diff -u sqlautocode/ my_sqlautocode/
 diff -u sqlautocode/config.py my_sqlautocode/config.py
 --- sqlautocode/config.py 2014-06-06 10:29:55.000000000 +0200
 +++ my_sqlautocode/config.py 2014-06-06 10:22:56.000000000 +0200
 @@ -140,7 +140,7 @@
 engine = sqlalchemy.create_engine(url)
 test = engine.connect()
 test.close()
 - except sqlalchemy.exceptions.SQLAlchemyError, ex:
 + except sqlalchemy.exc.SQLAlchemyError, ex:
 parser.error('Could not connect to "%s": %s' % (url, ex))
diff -u sqlautocode/declarative.py my_sqlautocode/declarative.py
 --- sqlautocode/declarative.py 2014-06-06 10:29:55.000000000 +0200
 +++ my_sqlautocode/declarative.py 2014-06-06 10:08:09.000000000 +0200
 @@ -14,8 +14,12 @@
 from sqlalchemy.ext.declarative import _deferred_relationship
 except ImportError:
 #SA 0.5 support
 - from sqlalchemy.ext.declarative import _deferred_relation as _deferred_relationship
 -
 + try:
 + from sqlalchemy.ext.declarative import _deferred_relation as _deferred_relationship
 + except ImportError:
 + #SA 0.8 support
 + from sqlalchemy.ext.declarative.clsregistry import _deferred_relationship
 +

Syntax highlighting for Kivy kv language in pycharm

10

The nice folks on the Kivy ML have provided a kv language highlighting for Pycharm ๐Ÿ™‚

For easy retrieval I just post the original announcement together with the settings file:

For anyone using PyCharm, please find attached a file type extension that give you full syntax highlighting for KV files. Courtesy of Xuton…;-)
To install:
On Pycharmโ€™s main menu, click File-> Import settings

ย ย ย  Select this file, then it will come with a dialog with filetypes ticked. Just click ok or whatever and then its there.

You can get the file here:

https://github.com/Zen-CODE/kivybits/blob/master/IDE/PyCharm_kv_completion.jar

 

Python xmlrpcserver with HTTPS support and basic authentication

6

For a project I’m working on at Formatics.nl I needed a Python based multithreaded xmlrpcserver with HTTPS support, HTTP basic authentication and capable of running on any port. After looking into solutions which required some sort of server for the HTTPS and basic autentication parts I decided to go for a better solution to try to add the HTTPS support and basic authentication to the Python xmlrpsserver itself. That way we would have a simple and fast solution which could run from any box on any port. Looking around on the net I found various solutions for every requirement but nothing which combined it all into one solution. So after combining various bits and pieces I came up with a Python xmlrpcserver with HTTPS and basic authentication support.

As the code used came from various free-software projects and locations I decided to share the server with the world so that it might benefit others.

For the HTTPS you need OpenSSL certificates, use your existing PEM formatted one or just create one yourself.

$ openssl genrsa -out privkey.pem 2048
$ openssl req -new -key privkey.pem -out cert.csr
$ openssl req -new -x509 -key privkey.pem -out cacert.pem -days 1095

And here’s the server

from SocketServer import ThreadingMixIn, BaseServer
from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler, SimpleXMLRPCDispatcher
import socket
from OpenSSL import SSL
from base64 import b64decode

from threading import Thread, Condition
from thread import start_new_thread

import traceback
import time

# static stuff
DEFAULTKEYFILE = 'privkey.pem'  # Replace with your PEM formatted key file
DEFAULTCERTFILE = 'cacert.pem'  # Replace with your PEM formatted certificate file

class Services:
    def give_me_time(self):
        return time.asctime()

class CustomThreadingMixIn:
    """Mix-in class to handle each request in a new thread."""
    # Decides how threads will act upon termination of the main process
    daemon_threads = True

    def process_request_thread(self, request, client_address):
        """Same as in BaseServer but as a thread.
        In addition, exception handling is done here.
        """
        try:
            self.finish_request(request, client_address)
            self.close_request(request)
        except (socket.error, SSL.SysCallError), why:
            print 'socket.error finishing request from "%s"; Error: %s' % (client_address, str(why))
            self.close_request(request)
        except:
            self.handle_error(request, client_address)
            self.close_request(request)

    def process_request(self, request, client_address):
        """Start a new thread to process the request."""
        t = Thread(target=self.process_request_thread, args=(request, client_address))
        if self.daemon_threads:
            t.setDaemon(1)
        t.start()

class MyXMLRPCServer(CustomThreadingMixIn, SimpleXMLRPCServer):
    def __init__(self, ip, port, keyFile=DEFAULTKEYFILE, certFile=DEFAULTCERTFILE, logRequests=True):
        self.logRequests = logRequests
        class VerifyingRequestHandler(SimpleXMLRPCRequestHandler):

            def setup(myself):
                myself.connection = myself.request
                myself.rfile = socket._fileobject(myself.request, "rb", myself.rbufsize)
                myself.wfile = socket._fileobject(myself.request, "wb", myself.wbufsize)

            def address_string(myself):
                "getting 'FQDN' from host seems to stall on some ip addresses, so... just (quickly!) return raw host address"
                host, port = myself.client_address
                #return socket.getfqdn(host)
                return host

            def do_POST(myself):
                """Handles the HTTPS POST request.
                It was copied out from SimpleXMLRPCServer.py and modified to shutdown the socket cleanly.
                """
                try:
                    # get arguments
                    data = myself.rfile.read(int(myself.headers["content-length"]))
                    # In previous versions of SimpleXMLRPCServer, _dispatch
                    # could be overridden in this class, instead of in
                    # SimpleXMLRPCDispatcher. To maintain backwards compatibility,
                    # check to see if a subclass implements _dispatch and dispatch
                    # using that method if present.
                    response = myself.server._marshaled_dispatch(data, getattr(myself, '_dispatch', None))
                except Exception, info: # This should only happen if the module is buggy
                    print "ERROR do_POST: ", info
                    print "Traceback follows:", traceback.print_exc()

                    # internal error, report as HTTP server error
                    myself.send_response(500)
                    myself.end_headers()
                else:
                    # got a valid XML RPC response
                    myself.send_response(200)
                    myself.send_header("Content-type", "text/xml")
                    myself.send_header("Content-length", str(len(response)))
                    myself.end_headers()
                    myself.wfile.write(response)

                    # shut down the connection
                    myself.wfile.flush()
                    myself.connection.shutdown() # Modified here!

            def do_GET(myself):
                """Handles the HTTP GET request.

                Interpret all HTTP GET requests as requests for server
                documentation.
                """
                # Check that the path is legal
                if not myself.is_rpc_path_valid():
                    myself.report_404()
                    return

                response = myself.server.generate_html_documentation()
                myself.send_response(200)
                myself.send_header("Content-type", "text/html")
                myself.send_header("Content-length", str(len(response)))
                myself.end_headers()
                myself.wfile.write(response)

                # shut down the connection
                myself.wfile.flush()
                myself.connection.shutdown() # Modified here!

            def report_404(myself):
                # Report a 404 error
                myself.send_response(404)
                response = 'No such page'
                myself.send_header("Content-type", "text/plain")
                myself.send_header("Content-length", str(len(response)))
                myself.end_headers()
                myself.wfile.write(response)
                # shut down the connection
                myself.wfile.flush()
                myself.connection.shutdown() # Modified here!

            def parse_request(myself):
                if SimpleXMLRPCRequestHandler.parse_request(myself):
                    basic, foo, encoded = myself.headers.get('Authorization').partition(' ')
                    username, foo, password = b64decode(encoded).partition(':')
                    #print username, foo, password
                    if username == 'admin':
                        return True
                    else:
                        myself.send_error(401, 'Authentication failed')
                        return False
        SimpleXMLRPCDispatcher.__init__(self, False, None)
        BaseServer.__init__(self, (ip, port), VerifyingRequestHandler)

        # SSL socket stuff
        ctx = SSL.Context(SSL.SSLv23_METHOD)
        ctx.use_privatekey_file(keyFile)
        ctx.use_certificate_file(certFile)
        self.socket = SSL.Connection(ctx, socket.socket(self.address_family, self.socket_type))
        self.server_bind()
        self.server_activate()

        self.funcs = {}
        self.register_introspection_functions()
        self.register_instance(Services())

        # requests count and condition, to allow for keyboard quit via CTL-C
        self.requests = 0
        self.rCondition = Condition()

    def startup(self):
        #run until quit signaled from keyboard
        print 'server starting; hit CTRL-C to quit...'
        while True:
            try:
                self.rCondition.acquire()
                start_new_thread(self.handle_request, ()) # we do this async, because handle_request blocks!
                while not self.requests:
                    self.rCondition.wait(timeout=3.0)
                if self.requests:
                    self.requests -= 1
                self.rCondition.release()
            except KeyboardInterrupt:
                print "quit signaled, i'm done."
                return

    def get_request(self):
        request, client_address = self.socket.accept()
        self.rCondition.acquire()
        self.requests += 1
        self.rCondition.notifyAll()
        self.rCondition.release()
        return (request, client_address)

    def listMethods(self):
        """return list of method names (strings)"""
        methodNames = self.funcs.keys()
        methodNames.sort()
        return methodNames

    def methodHelp(self, methodName):
        """method help"""
        if methodName in self.funcs:
            return self.funcs[methodName].__doc__
        else:
            raise Exception('method "%s" is not supported' % methodName)

if __name__ == '__main__':
    import sys
    if len(sys.argv) == 3:
        host = sys.argv[1]
        port = int(sys.argv[2])
    else:
        host = 'localhost'
        port = 8111

    server = MyXMLRPCServer(host, port, logRequests=True)
    # Run the server's main loop
    sa = server.socket.getsockname()
    print "Serving HTTPS on", sa[0], "port", sa[1]

    server.startup()

For a client use someting like this

from xmlrpclib import ServerProxy

user = 'stas'
pas = 'stas'
p = ServerProxy('https://%s:%s@localhost:8111' % (user, pas))
print p.give_me_time()

Pycharm exclude files from beeing synchronized

0

When running on a slower system or laptop synchronizing can take some time when you have a lot of binary files. The solution is to exclude them.

Settings -> Deployment -> Exclude items by name:
For example:
.svn;.cvs;.idea;.DS_Store;.git;.hg;*/DbaseAssets/Images/*;*/DbaseAssets/Sounds/*;*/builder/*/App

Building a kivy app for Android issues

0

When making a kivy app for Android I came across some issues and while I informed the kivy devs I just put them here also just in case I need them later ๐Ÿ™‚

Just follow the manual for packaging for Android.
First of all when you install the dependencies you must use python-pip iso pip.
When building the python binary I had problems with the httplib module which did have https support.
This what adb logcat told me:
I/python (19347): your version of httplib doesn’t support HTTPS

As it turned out the order in which you build the various modules matter.
So when building the Python binary you must use the order.
Put C lib first, then python lib, then kivy.
eg ./distribute.sh -m “sqlite3 libxml2 pyopenssl pil kivy”

Now there’s HTTPS support.

When you still haven’t HTTPS support try to clean the python4android git tree.
git clean -dxf

More android specific stuff I encountered:
To write to the filesystem use CWD/.kivy/….
sys.platform is on android ‘linux3’ (GNU/Linux is ‘linux2’)

Excluding files in git and egit on eclipse

0

This is not rocket science but more a reminder for myself ๐Ÿ™‚

When working with eclipse and the egit plugin I noticed that a ‘git refresh’ would take minutes in certain projects. The particular project has a few directories containing 10000+ .ogg and .png files which are not included in the git repo. I have git exclude files and global .gitignore files in place but egit seems to ignore them.
So I added the file patterns to the egit ‘Ignored Recources’.

(For example I want to exclude everything in my */lib/Sounds)
Open Window -> Preferences -> Team -> EGit -> Add pattern
The pattern to add: */lib/Sounds/* (duhhh)
Push ‘apply’ and presto, the refresh just takes seconds ๐Ÿ™‚

Just for completeness I include my .gitignore_global, the relevant parts and the projects exclude file.
To have the git global ignoring stuff do: git config --global core.excludesfile ~/.gitignore_global

.gitignore_global:
# Custom stuff
*.project
*.pydevproject
*.settings
*.pyc
….
….

The .git/info/exclude file:
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~
builder/BtpOS/*/Data/
lib/CPData/DbaseAssets/Images/
lib/CPData/DbaseAssets/Sounds/

youtube_pl2vids.py

0

Python script to download youtube vids

[sourcecode language=”python”]

import sys, os, random, time

if len(sys.argv) < 3:
print "Usage: youtube_pl2vids [playlist.html] [destination directory]"
sys.exit(1)

source = sys.argv[1]
dest = sys.argv[2]
urllines = []
f = open(os.path.join(dest, ‘youtube.dl’), ‘w’)
for line in open(source, ‘r’).readlines():
if ‘<a href="/watch?v=’ in line and ‘class="tile-link-block video-tile"’ in line:
url = "http://www.youtube.com" + line.split(‘&amp’)[0].split(‘<a href="’)[1]
f.write(url + ‘n’)
urllines.append(url)
f.close()
print urllines
# we assume youtube-dl is installed
for line in urllines:
os.system(‘youtube-dl -t –max-quality 22 %s’ % line)
s = random.randint(1, 6)
print "sleeping %s secs" % s
time.sleep(s)
[/sourcecode]

Go to Top