Search and test all links available in web page

Another small utility to check if all HYPERlinks inside a web page are working operationally. User can modify this utility and use according to their requirement.The nice part of this code is it stores search output in a file and display storage path on GUI console at the end of the process. To test the script simply store the code in a *.py file and double click on it. A popup will appear on desktop. Provide URL/web page and click on "show" button.
Searching will start and result will populate on windows command line console provided, python-3.3.2 is installed and placed in path properly.

#!python
'''
Created on Aug 19, 2013
@requires: Tested in Windows XP
@author: Jaydeb Chakraborty
@version: Python Version-3.3.2

'''
from tkinter import Tk, Label, Button, Entry
import urllib.request
import re
import os
import logging

""" 
Store all URLs in a file and get HTTP return code 
"""
def show():
    strn = entry.get()  
    if re.match('(?:ftp|https)://', strn):
        mesg="Currently HTTPS|FTP is not supported"
        t = Label(w, text=mesg)
        t.pack()
    else:
        l=strn.replace('http://', '')
        mesg='Please check output in ' + (os.environ.get('TEMP', '')) + '\INFO.log'
        t = Label(w, text=mesg)
        t.pack()                
         
    if strn:  
        stdinfo=((os.environ.get('TEMP', ''))+ '\INFO.log')
        logging.basicConfig(filename='%s' % stdinfo, format='%(asctime)s %(message)s', level=logging.INFO)
        logging.warning('*******  Accessing : %s' % strn + ' ********')
        logging.warning('**********************************************')
        local_filename, headers = urllib.request.urlretrieve('http://' + l)
        f = open(local_filename) 
        for lines in f:
            myString_list = [item for item in lines.split(" ")]
            for item in myString_list:      
                try:
                    o = re.search("(?Phttp?://[^\s]+)", item.expandtabs()).group("url")
                    url = re.sub(r'\?.*|".*', "", o)
                    conn = urllib.request.urlopen(url)
                    access = conn.getcode()
                    logging.warning('URL : %s' % url + ' -- Returncode is : %s' % access)
                    print(url, access)
                except :
                    pass 
                
       
w = Tk()
quitBotton = Button(w, text='Quit', command=quit).pack()
showBotton = Button(w, text='Show', command=show).pack()
Label(w, text="        Please provide URL...        ").pack()
entry = Entry(w)
entry.pack()
res = Label(w)
res.pack()
w.title('Test Links in web page')
w.maxsize(1000, 40000)
w.mainloop()

Windows schedule task: Restart Tomcat automatically

Managing tomcat with a bunch of application sometime becomes problematic. Without investing proper time for root cause analysis it is not possible to isolate the issue perhaps the application from the container which is troubling. Be it a production environment project cannot move forward w farther without resolving the issue. Well, I am talking about development environment where usually we do not think about the impact instead just bounce the server.

For UNIX/Linux it is easy at least can be managed using a few line of script and scheduling the job via cron. Maybe windows expert will do the same very easily. I thought to do the same thing via python. There are many reason and corresponding indication of hanging a server. Having a close look in server log one can found the common foot print that a server left every time it goes to hang state. High Memory utilization is one of them and very common.  Following steps demonstrate how to setup a job written in python in windows.

How to schedule jobs in windows?
>>schtasks /create /tn "PythonCron Job" /tr "path/to/your/executable Arg1 Arg2" /sc hourly
Above command will create job in "Control Panel--> Schedule Task".
In windows, schedule python script does not accept argument.
Therefore converted .py script to .exe file

How to convert .py to .exe?
To Convert .py to .exe for Python3.3 I used CX-Freeze.
For more details please have a look in http://cx-freeze.sourceforge.net/index.html
>>cxfreeze hello.py --target-dir dist
.........and schedule task as
>>C:\Path\To\Your\File.exe Arg1 Arg2

How to adjust schedule job in windows?
Open the task and adjust time in Advance Tab. Every task requires a user name and password
Administrator can use "NT AUTHORITY\SYSTEM"
How to check log?
To check if schedule job is running perfectly, go to "Control Panel--> Schedule Task"
Logs are usually available in "C:\WINDOWS\SchedLgU"
The script also populates logs ‘stdout/stderror’ in user "HOMEPATH". You can check logs if there is any error.
How to check login user in windows
>>echo %USERDOMAIN%\%USERNAME%

How to Test script in DEV?
To test this script please follows the following steps:
    1) Defile CATALINA_HOME in User Variable, if not defined.
    2) Start tomcat server
    3) Execute script in command line "python /path/to/the/script.py MAXMEM TOMCAT_PORT"
    4) For testing provide MAXMEM as low as for example '5'
        Example: >>>python /path/to/the/script.py 5 8080

#!python
'''
Python Version-3.3.2
Created on Aug 19, 2013
@author: Jaydeb Chakraborty
??? Restart Tomcat automatically
??? Monitor server behavior and modify script for different threshold 

!!! Assume CATALINA_HOME is defined and Tomcat starts via 'startup.bat'
'''
import logging
import sys
import re
import os
import socket
import subprocess
import urllib.request
from datetime import datetime
from time import sleep

class ServerProcess:
    def __init__(self, portnum):
        self.portnum = portnum
        self.port = (str(portnum).replace("'", "").replace("]", ""))
    def getPID(self):
        try:
            port = self.port
            p1 = subprocess.Popen('netstat -ano', stdout=subprocess.PIPE)
            p2 = subprocess.Popen('findstr :%s' % port, stdin=p1.stdout, stdout=subprocess.PIPE)
            result=p2.communicate()[0].split()
            pid=str(result[4]).lstrip("b,").replace(",", "").replace("'", "")
            return pid 
        except LookupError as l:
            logging.basicConfig(filename='%s' % stderror, format='%(asctime)s %(message)s', level=logging.DEBUG)
            logging.warning('Watch out! : %s' % l)
            return 0       
    def getMemory(self, pid):
            p3 = subprocess.Popen('tasklist /fi "MEMUSAGE ge 10" /fi "PID eq %s' % pid, stdout=subprocess.PIPE)
            mem = str(p3.communicate()[0].split()[17]).lstrip("b,").replace(",", "").replace("'", "")
            return mem
    def getConnect(self):
        try:
            port = int(self.port)
            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            remoteServerIP  = socket.gethostbyname(socket.gethostname())
            conn = sock.connect((remoteServerIP, port))
            sock.close()
            return 'Listening'
        except socket.gaierror as name:
            logging.basicConfig(filename='%s' % stderror, format='%(asctime)s %(message)s', level=logging.DEBUG)
            logging.warning('Watch out! : %s' % name)
            sys.exit()
        except socket.error as s:
            logging.basicConfig(filename='%s' % stderror, format='%(asctime)s %(message)s', level=logging.DEBUG)
            logging.warning('Watch out! : %s' % s)
            return 'NotListening'
            sys.exit()
    def getHttpCode (self):
        port = self.port
        req=('http://localhost:'+ (str(Args[2]).replace(",", "").replace("'", "").replace("]", "")))
        status = urllib.request.urlopen(req)
        data=status.getcode()
        return data        
    def setLogentries(loglevel):
        if (loglevel == 'DEBUG'):
            logging.basicConfig(filename='%s' % stderror, format='%(asctime)s %(message)s', level=logging.DEBUG)
        else:
            logging.basicConfig(filename='%s' % stdout, format='%(asctime)s %(message)s', level=logging.INFO)

if __name__ == "__main__":
    global stderror, stdout 
    stdout=((os.environ.get('HOMEPATH', ''))+ '\INFO.log')
    stderror=((os.environ.get('HOMEPATH', ''))+ '\ERROR.log')
    Args = str(sys.argv).split()
    # Check Arguments
    if ((len(Args)) != 3):
        ServerProcess.setLogentries('DEBUG')
        logging.warning('Watch out! : Provide Hostname and Port name in arguments')
        print("Uses : python " + str(Args[0]).replace("['", "") + " MAXMEM PORT")
        sys.exit()

maxmem=str(Args[1]).replace(",", "").replace("'", "")        
process = ServerProcess(Args[2])
pid = process.getPID()
memory = process.getMemory(pid)      

for i in range(3):
    sleep(1)
    sys.stdout.flush()
    conn = process.getConnect()
    data = process.getHttpCode()
    """ If socket is listening and used memory is less than MAXMEM """
    if(( conn == 'Listening' ) and eval(maxmem)>eval(memory)  and ( data == 200 )):
        ServerProcess.setLogentries('INFO')
        logging.warning('Watch out! : Socket is %s ' % conn + 'when used memory is %s ' % memory + 'and return code is %s ' % data)
    elif(eval(maxmem) > eval(memory) and data == 200):
        ServerProcess.setLogentries('DEBUG')
        logging.warning('Watch out! : Can access url')              
    else:
        p = subprocess.Popen('tasklist /fi "MEMUSAGE ge 10" /fi "PID eq %s' % pid, stdout=subprocess.PIPE)
        proc = str(p.communicate()[0].split()[13]).lstrip("b,").replace(",", "").replace("'", "")
        os.system('taskkill /f /im %s' % proc)
        try:
            startup=(os.environ.get('CATALINA_HOME', '')+'\/bin\/startup.bat')
            os.system('%s' % startup)
            sys.exit()
        except OSError as sys:
            ServerProcess.setLogentries('DEBUG')
            logging.warning('Watch out! : Socket is %s' % sys)