Mapping point locations with Python and Microsoft Live Search Maps

Chad — Wed, 16 Apr 2008 03:04:43 +0000

This article was first published in the February 2008 issue of Python Magazine . It is being reproduced here with the permission of Marco Tabini and Associates .

Requirements

Python 2.4+
pyExcelerator module (http://sourceforge.net/projects/pyexcelerator/ )
Anthony Tuininga’s ceODBC module (http://sourceforge.net/html/ceODBC.html )
John Machin’s xlrd module (http://www.lexicon.net/sjmachin/xlrd.htm )
Michael Foord’s cgiutils module (http://www.voidspace.org.uk/python/cgiutils.html )

Lets face it, using Python, we can pull data from virtually anywhere in the corporate enterprise – databases, text files, or other documents. But how can we easily visualize data that has place associated with it? Well, we turn to Python and the API of a popular web-mapping application, that’s how.

I was recently tasked with finding a way to routinely (read daily) present a dataset of point locations on a onscreen map to our team of 40 or so people. I needed a way to easily disseminate some sort of map product that would display the data, something that required no special software setup on PCs – and most importantly – something that would be easy and intuitive for our team members to use. I thought about providing a PDF map product – I would have to harvest the data, process it, plot it on a map using mapping software, and email the final product to my team members as an attachment. Doesn’t sound like much fun, does it? Oh, and did I mention that those would all be manual processes to be completed once per day? While pondering this not-so-desirable methodology, I ran across a clever method used by our Information Technology team that passed locational data in a URL to Microsoft Live Maps. Their process takes point locations (in the form of latitude and longitude) from SQL Server, and using VB.Net and the Microsoft Live Search Maps URL API, builds a URL that plots well locations on a Live Search Maps web map. This was it, I would harvest the data, process it, build my URL using the Live Search Maps URL API, and send out the URL to my team nightly in a email. And I would do all of this using Python.

Example dataset

The dataset I needed to map is a gas well drilling schedule stored in a Microsoft Office Excel 2003 workbook. Rows in the workbook represent natural gas wells on the schedule and each has attributes associated with it such as well name, latitude, longitude, and the date drilling commenced; or spud date, as it is called in the oil and gas industry. The schedule gets updated at least daily, and in this line of business, plans can change every hour – literally. However, it was decided that a process that mapped the locations of the wells on a daily basis would suffice. Also note that, as will be demonstrated later, you don’t necessarily need point locations with latitude and longitude to benefit from this process; address locations can be used also.

Choosing a web mapping application

When first exploring the option of using a existing web mapping service to plot my well locations, I looked into most of the popular services out there: Google Maps (http://maps.google.com/, my personal favorite for the up-to-date aerial photography and ability to save your own maps, among other things), Yahoo! Maps (http://maps.yahoo.com/), and Microsoft Live Search Maps (http://maps.live.com/ – Figure 1). As stated earlier, our IT team had chosen Live Search Maps for an earlier application, and I soon learned the reason in doing so is that at the time, Live Search Maps was the only service out there which provided the ability to pass in multiple point locations to plot via latitude and longitude through the URL string. This was very important to the process, as I want to simply pass in all of my well locations through the URL, and then provide this URL to my team members.

Features of Microsoft Live Search Maps

Aside from the already-discussed feature of passing in multiple locations through the URL string, another great feature of Live Search Maps is the Scratch Pad – think of it as a "table of contents" for your map (Figure 2). The URL API (which will be discussed shortly) provides us the ability to create a scratch pad which lists each item and a description in a list on the right side of our map. Users can then see a scrollable list of all items that are currently plotted on the map. Also included in the Scratch Pad is the ability to zoom to street, city, or regional levels, send a location in an email, or obtain driving directions to or from a location. No other mapping service was able to provide these capabilities. So using Live Search Maps, my team members would be provided with a somewhat "traditional" web mapping interface that would be easy and intuitive to use, and with the Scratch Pad, they would be given a list of the mapped locations as well as the point locations plotted on the map.

Microsoft Live Search Maps URL API

The Live Search Maps Online Help (http://help.live.com/Help. aspx?market=en-US&project=WL_Local&querytype=topic&query=WL_ LOCAL_PROC_BuildURL.htm ) provides us with full documentation to the Live Search Maps URL API. I won’t go over everything, but I will cover what parts this process uses to build a URL – you can read the rest at your leisure. The API is fairly simple and straightforward. The first step is to build your standard Live Search Maps URL base – it’s always the same:

http://maps.live.com/default.aspx?

Next, you start to add your parameters. For us, that means set the map style. There are four options to choose from: aerial photo (a), road map (r), hybrid – aerial photo with labels and some roads (h), and oblique – commonly referred to as "bird’s eye view" (o). For this application I decided to use the road map view, since it would provide towns, cities, and major roads – all labeled – for users to orient themselves by:

http://maps.live.com/default.aspx?style=r

Next, we start to define and add our custom annotations to pass in and add to the Scratch Pad. For annotations, the form is:

sp=an.latitude_longitude_title_description

Latitude and longitude are obviously the locational information for the well, and you are given the option of adding a title and description (they end up on different lines in the scratch pad). So our URL now becomes something like so for the USA 1-18 well set to spud on 11/12/2007 and drilled by rig Smith #2 (note the encoding values used for the dash in the well name and the pound symbol in the rig name – don’t forget to encode all of your special characters in your URL string):

http://maps.live.com/default.aspx?style=r&sp=an.35.265638147_-93.5877380077_USA%201-18_11/12/2007%20-%20Smith%202

Basically, thats it! Copy and paste the URL above into your web browser and you will get a Live Search Maps map with the USA 1-18 well on the map and in the Scratch Pad. To add additional point locations to the URL, simply add a tilde and the next custom annotation(s):

http://maps.live.com/default.aspx?style=r&sp=an.35.265638147_-93.5877380077_USA%201-18_11/12/2007%20-%20Smith %202~an.35.0878763351_-93.5096775211_Jones%2013-13_11/01/2007%20-%20Smith%2035

Harvesting and processing the Excel data

Before we can build our map, we have to get the data out of our Excel workbook. As Figure 3 shows, the worksheet layout is fairly simple. The columns represent, from left to right, well name, spud date (date the well started drilling), total depth, or TD date (the date the well finished drilling and reached total depth), latitude, longitude, and drilling rig number. To harvest our data from the workbook, we will enlist the help of the PyExcelerator module (http://sourceforge.net/projects/pyexcelerator ) and a function that comes with the docs – which worked right out of the box! Essentially we read the Excel workbook and return a list of lists, where each nested list represents a row in our worksheet. The entire function for parsing the Excel workbook can be found in Listing 1.

import pyExcelerator, os, time, string, calendar, datetime
import smtplib, cgiutils, xldate, DateTime, EmailRecips
def ExcelToList(fname, worksheet=0, encoding='cp1251'):
data = pyExcelerator.parse_xls(fname,encoding)
sheet_name=data[worksheet][0].encode(encoding)
values = data[worksheet][1]
for each in sorted(data[0][1].items()):
print str(each)[1:-1]
vdict = {}
row_idx_max = 0
col_idx_max = 0
for row_idx, col_idx in sorted(values.keys()):
row_idx_max = max(row_idx,row_idx_max)
col_idx_max = max(col_idx,col_idx_max)
v = values[(row_idx, col_idx)]
if isinstance(v, unicode):
v = v.encode(encoding, 'backslashreplace')
vdict[(row_idx,col_idx)] = v
vlist = []
for row in range(row_idx_max+1):
vlist.append([])
for col in range(col_idx_max+1):
if (row,col) not in vdict:
vdict[(row,col)]=None
vlist[row].append(vdict[(row,col)])
return vlist
Download this code: listing1.txt

The first step in ExcelToList utilizes the parse_xls method of pyExcelerator, which returns a list of two-tuples; one tuple for each worksheet in the workbook where each tuple consists of the worksheet name and worksheet data. The worksheet data is a dictionary mapping each non-blank cell to its cell value:

import pyExcelerator, os, time, string, calendar, datetime, import smtplib, cgiutils, xldate, DateTime data = pyExcelerator.parse_xls(filename, encoding)

Which returns:

[('Sheet1', {(0, 1): 39412.0, (1, 2): 39437.0, (2, 4): -93.600387566799995, (0, 0): 'Smith #2-3', (2, 0): 'Coop #4-17', (2, 2): 39452.0, (1, 4): -94.856870378300002, (1, 1): 39421.0, (1, 5): 'Rig 4', (1, 3): 34.831162240499999, (0, 5): 'Rig 67', (2, 1): 39431.0, (2, 5): 'Rig 23', (0, 4): -97.340309266600002, (1, 0): 'Jones #1-19', (2, 3): 35.232594439099998, (0, 3): 35.276017337600003, (0, 2): 39430.0}), ('Sheet2', {}), ('Sheet3', {})]

Note that we returned a three-item list, but the dictionaries for Sheets 2 and 3 are empty, as there is no data in those worksheets. From this we get the dictionary for the worksheet we want out of the list data (where worksheet=0 ):

values = data[worksheet][1]

Which returns:

{(0, 1): 39412.0, (1, 2): 39437.0, (2, 4): -93.600387566799995, (0, 0): 'Smith #2-3', (2, 0): 'Coop #4-17', (2, 2): 39452.0, (1, 4): -94.856870378300002, (1, 1): 39421.0, (1, 5): 'Rig 4', (1, 3): 34.831162240499999, (0, 5): 'Rig 67', (2, 1): 39431.0, (2, 5): 'Rig 23', (0, 4): -97.340309266600002, (1, 0): 'Jones #1-19', (2, 3): 35.232594439099998, (0, 3): 35.276017337600003, (0, 2): 39430.0}

Sorted, it’s much easier to see what’s going on:

for each in sorted(data[0][1].items()): print str(each)[1:-1]

(0, 0), 'Smith #2-3' (0, 1), 39412.0 (0, 2), 39430.0 (0, 3), 35.276017337600003 (0, 4), -97.340309266600002 (0, 5), 'Rig 67' (1, 0), 'Jones #1-19' (1, 1), 39421.0 (1, 2), 39437.0 (1, 3), 34.831162240499999 (1, 4), -94.856870378300002 (1, 5), 'Rig 4' (2, 0), 'Coop #4-17' (2, 1), 39431.0 (2, 2), 39452.0 (2, 3), 35.232594439099998 (2, 4), -93.600387566799995 (2, 5), 'Rig 23'

Here again, keys represent the cell mappings (Excel cell A1 = (0,0) ) and values represent the data within the cell. Next, we iterate through our values object, putting our key/value cell reference/cell value pairs into the vdict object and getting the maximum number of rows and colums (lines 10-19, Listing 1):

Here, our max is 2 rows and 5 columns, since it is zero- based. Finally, we iterate through our vdict object, converting it to a two-dimensional list with each list item representing a row in our Excel worksheet. Blank cells are represented by None :

vlist = [] for row in range(row_idx_max+1): vlist.append([]) for col in range(col_idx_max+1): if (row,col) not in vdict: vdict[(row,col)]=None vlist[row].append(vdict[(row,col)])

Our end product returned from ExcelToList looks like so:

[['Smith #2-3', 39412.0, 39430.0, 35.276017337600003, -97.340309266600002, 'Rig 67'], ['Jones #1-19', 39421.0, 39437.0, 34.831162240499999, -94.856870378300002, 'Rig 4'], ['Coop #4-17', 39431.0, 39452.0, 35.232594439099998, -93.600387566799995, 'Rig 23']]

Now we have a nice list that we can simply iterate through and process later.

But Wait a Minute! My Data is in ! How do I get it out to use for something like this?

Never fear, if your data is in a RDBMS, you’re actually better off. If you don’t already have it, head on over to SourceForge and pick up Anthony Tuininga’s great ODBC module ceODBC (http://ceodbc.sourceforge.net/html/ceODBC.html ) that will allow you to connect to your database. I tested it on SqlServer and DB2 with great success. Listing 2 demostrates how to use ceODBC with SqlServer.

>>> import ceODBC
>>> db=ceODBC.connect('DSN=Wells')
>>> c=db.cursor()
>>> c.execute('select wellname, latitudedecimal,
... longitudedecimal from dbo.wells where wellid in
... (11587,11194,11157)')
>>> for each in c.fetchall():
... print each
...
('GILL #1-18', 33.095599, -92.38563)
('HOW #2-7', 35.10155, -91.48824)
('JKK #11-13', 34.09130, -93.45256)
Download this code: listing2.txt

This method employs a User DSN I created in the Windows ODBC Administator which uses my Windows credentials to log on to SqlServer. Of course, there are a plethora of database connectivity modules out there, so use whichever you want.

Building the Live Search Maps URL

Now that we have all of our data from the Excel workbook (or RDBMS), we can use it to build Scratch Pad annotations and eventually our URL. Based on the URL API, this looks easy enough, but let’s perform a few other litle tricks that will make things display much nicer in the Scratch Pad. First, we need to set our URL base:

urlBase = 'http://maps.live.com/default.aspx?style=r'

Now we need to iterate through our well data list from our Excel workbook, place the list items into namespaces and perform some processing on some of them, then build a annotation string from each one. There are two functions we should discuss first. You may have noticed earlier that after we slurp our data out of our Excel workbook, we are left with date values that look like 39430.0 and 39452.0 . Huh? Many of you will recognize these as Excel dates. Excel stores dates as floating point numbers; the number of days (or fraction thereof) since 12/31/1899, to be exact. For example, 9/20/2006 is actually 38980 to Excel; Excel just formats it to look like a date for simplicity of viewing. To deal with Excel dates and turn them into something I could actually use, I turned to the xlrd module (http://www.lexicon.net/sjmachin/xlrd.htm ) and its function xldate_as_tuple. Also, to get the well status, we assume that anytime today’s date falls within the window greater than or equal to spud date (beginning of drilling) and less than or equal to td date (end of drilling), that well is currently drilling. We’d like to know this and denote it in the Scratch Pad, so we determine each wells status. And if indeed the well is drilling, we note it in our annotation (Listing 3).

def GetWellStatus(inSpudDt, inPrjTdDt):
if len(inSpudDt) > 0 and len(inPrjTdDt) > 0:
today1 = time.strftime('%m/%d/%Y',time.localtime())
today = (datetime.date(int(today1.split('/')[2]),
int(today1.split('/')[0]),
int(today1.split('/')[1])))
spudDt = (datetime.date(inSpudDt[0],
inSpudDt[1],inSpudDt[2]))
prjTdDt = datetime.date(inPrjTdDt[0],inPrjTdDt[1],
inPrjTdDt[2])
if today > prjTdDt:
status = 'Drilled'
elif today >= spudDt and today <= prjTdDt:
status = 'Drilling'
elif today < spudDt:
status = 'Proposed'
else:
status = 'Unknown'
else:
status = 'Undeterminable'
return status
def KillObject(toBeKilled):
if os.path.exists(toBeKilled):
os.remove(toBeKilled)
def WriteOutput(file, text):
writer = open(file, 'a')
writer.write(text)
def BuildLiveMapsUrl(wells, ds):
urlBase = 'http://maps.live.com/default.aspx?style=r'
wholeList = []
dict = {}
for well in wells:
eachList = []
name = (well[0].replace(' ','%20')).replace('#','%23')
spudDateTuple = xldate.xldate_as_tuple(well[1],0)
spudDate = str(spudDateTuple[0])+'/'+str(spudDateTuple[1])+'/'+str(spudDateTuple[2])
tdDateTuple = xldate.xldate_as_tuple(well[2],0)
tdDate = str(tdDateTuple[0])+'/'+str(tdDateTuple[1])+'/'+str(tdDateTuple[2])
lat = well[3]
long = well[4]
rig = well[5].replace(' ','%20')
status = GetWellStatus(spudDateTuple, tdDateTuple)
if status == 'Drilling':
eachList.append('an.' + str(lat) + '_' + str(long) + '_' + name + '_' + spudDate +
'%20-%20' + rig + '%20-%20' + status.upper)
else:
eachList.append('an.' + str(lat) + '_' + str(long) + '_' + name + '_' + spudDate +
'%20-%20' + rig)
eachList.append(spudDate)
wholeList.append(eachList)
wholeList.sort(lambda x,y:cmp(x[1],y[1]))
for whole in wholeList:
print whole
urlList = []
for each in wholeList:
del each[1]
wellString = str(each)[2:len(each)-3]
urlList.append(wellString)
urlStringWells = string.join(urlList,'~')
masterUrl = urlBase + '&sp=' + urlStringWells + '~an.36.1164211465469_-94.143074863021_SEECO%
20Fayetteville%20Office~an.35.513123_-93.834189_SWN%20Ozark%20Office~an.35.0685432008022_
-92.4189197874021_SWN%20Conway%20Office~an.'
f = 'C:/temp/DS_Html_Email.html'
t = 'C:/temp/DS_Text_Email.txt'
dsUpdated = str(time.strftime('%A, %m/%d/%Y, %I:%M %p', time.localtime(os.path.getmtime(ds))))
KillObject(f)
KillObject(t)
WriteOutput(f,'\n\n




AAT Drill Schedule Mapper for Friday, 11/23/2007


Drill schedule last updated: Friday, 11/23/2007, 10:08 PM


 


Smith%20%232-3_2007/11/21%20-%20Rig%2067%20-%20DRILLING~an.35.2325944391_-93.6003875668_
Coop%20%234-17_2007/12/15%20-%20Rig%2023~an.34.8311622405_-94.8568703783_Jones%20%231-19_
2007/12/5%20-%20Rig%204~an.36.1164211465469_-94.143074863021_SEECO%20Fayetteville%20Office
~an.35.513123_-93.834189_SWN%20Ozark%20Office~an.35.0685432008022_-92.4189197874021_SWN%20
Conway%20Office~an.">Click here for a map of the latest AAT drill schedule


 


*** THIS MESSAGE AUTO-GENERATED ***


*** PLEASE DO NOT REPLY TO THIS EMAIL ***




Download this code: listing4.txt


Our text version is much simpler:
WriteOutput(t,'AAT Drill Schedule Mapper for ' + str(time.strftime('%A, %m/%d/%Y',time.localtime())))

WriteOutput(t,'\n\nDrill schedule last updated: ' + dsUpdated)

WriteOutput(t,'\n\nCopy and paste this address into your web broswer:\n\n' + masterUrl)

WriteOutput(t,'\n\n** This message auto-generated - please do not respond to this email ***') 
Which provides us with text output of:
AAT Drill Schedule Mapper for Friday, 11/23/2007
Drill schedule last updated: Friday, 11/23/2007, 10:08 PM
Copy and paste this address into your web broswer:
http://maps.live.com/default.aspx?style=r&sp=an.35.2760173376_-97.3403092666_Smith%20%232-3_2007/11/21%20-

%20Rig%2067%20-%20DRILLING~an.35.2325944391_-93.6003875668_Coop%20%234-17_2007/12/15%20-%20Rig%

2023~an.34.8311622405_-94.8568703783_Jones%20%231-19_2007/12/5%20-%20Rig%204~an.36.1164211465469_

-94.143074863021_SEECO%20Fayetteville%20Office~an.35.513123_-93.834189_SWN%20Ozark%20Office~an.

35.0685432008022_-92.4189197874021_SWN%20Conway%20Office~an.
*** This message auto-generated - please do not respond to this email ***
Disseminating the map to our team members
Our final step in this process is to send out to the team the email containing the URL to the day’s drill schedule. To do this I enlisted the help of Michael Foord’s cgiutils module (http://www.voidspace.org.uk/python/cgiutils.html ) and the createhtmlmail function (Listing 3). createhtmlmail takes our html and text files we just created and uses them as inputs to create a mime-message that will render as html or text, depending on the receiving email client. If no html is supplied, then the htmllib module is used to guess a text rendering. createhtmlmail returns us a message object which is passed into the SendEmail function; which also takes a list of email addresses of our team members as input. To send our email, we simply create a connection to our Outlook Exchange Server and pass in the sender, our list of recipients, and our html/text message to the smtplib.sendmail object. Finally, we terminate the SMTP session and close out the connection.
def SendEmail(message, recipients):

sender = 'AAT_Drill_Schedule_Mapper'

server = smtplib.SMTP(ip_address)

server.sendmail(sender, recipients, message)

server.quit()
And that’s it! I run this script as a Scheduled Task on Windows XP every night at 9PM, and a few minutes later, the email goes out to each of our team members (Figure 4). Listing 5 shows the process in its entirety.


import pyExcelerator, os, time, string, calendar, datetime
import smtplib, cgiutils, xldate, DateTime, EmailRecips
 
def ExcelToList(fname, worksheet=0, encoding='cp1251'):
data = pyExcelerator.parse_xls(fname,encoding)
sheet_name=data[worksheet][0].encode(encoding)
values = data[worksheet][1]
for each in sorted(data[0][1].items()):
print str(each)[1:-1]
vdict = {}
row_idx_max = 0
col_idx_max = 0
for row_idx, col_idx in sorted(values.keys()):
row_idx_max = max(row_idx,row_idx_max)
col_idx_max = max(col_idx,col_idx_max)
v = values[(row_idx, col_idx)]
if isinstance(v, unicode):
v = v.encode(encoding, 'backslashreplace')
vdict[(row_idx,col_idx)] = v
vlist = []
for row in range(row_idx_max+1):
vlist.append([])
for col in range(col_idx_max+1):
if (row,col) not in vdict:
vdict[(row,col)]=None
vlist[row].append(vdict[(row,col)])
return vlist
 
def GetWellStatus(inSpudDt, inPrjTdDt):
if len(inSpudDt) > 0 and len(inPrjTdDt) > 0:
today1 = time.strftime('%m/%d/%Y',time.localtime())
today = (datetime.date(int(today1.split('/')[2]),
int(today1.split('/')[0]),
int(today1.split('/')[1])))
spudDt = (datetime.date(inSpudDt[0],
inSpudDt[1],inSpudDt[2]))
prjTdDt = datetime.date(inPrjTdDt[0],inPrjTdDt[1],
inPrjTdDt[2])
if today > prjTdDt:
status = 'Drilled'
elif today >= spudDt and today <= prjTdDt:
status = 'Drilling'
elif today < spudDt:
status = 'Proposed'
else:
status = 'Unknown'
else:
status = 'Undeterminable'
return status
 
def KillObject(toBeKilled):
if os.path.exists(toBeKilled):
os.remove(toBeKilled)
 
def WriteOutput(file, text):
writer = open(file, 'a')
writer.write(text)
 
def BuildLiveMapsUrl(wells, ds):
urlBase = 'http://maps.live.com/default.aspx?style=r'
wholeList = []
dict = {}
for well in wells:
eachList = []
name = (well[0].replace(' ','%20')).replace('#','%23')
spudDateTuple = xldate.xldate_as_tuple(well[1],0)
spudDate = str(spudDateTuple[0])+'/'+str(spudDateTuple[1])+'/'+str(spudDateTuple[2])
tdDateTuple = xldate.xldate_as_tuple(well[2],0)
tdDate = str(tdDateTuple[0])+'/'+str(tdDateTuple[1])+'/'+str(tdDateTuple[2])
lat = well[3]
long = well[4]
rig = well[5].replace(' ','%20')
status = GetWellStatus(spudDateTuple, tdDateTuple)
if status == 'Drilling':
eachList.append('an.' + str(lat) + '_' + str(long) + '_' + name + '_' + spudDate +
'%20-%20' + rig + '%20-%20' + status.upper)
else:
eachList.append('an.' + str(lat) + '_' + str(long) + '_' + name + '_' + spudDate +
'%20-%20' + rig)
eachList.append(spudDate)
wholeList.append(eachList)
wholeList.sort(lambda x,y:cmp(x[1],y[1]))
for whole in wholeList:
print whole
urlList = []
for each in wholeList:
del each[1]
wellString = str(each)[2:len(each)-3]
urlList.append(wellString)
urlStringWells = string.join(urlList,'~')
masterUrl = urlBase + '&sp=' + urlStringWells + '~an.36.1164211465469_-94.143074863021_SEECO%
20Fayetteville%20Office~an.35.513123_-93.834189_SWN%20Ozark%20Office~an.35.0685432008022_
-92.4189197874021_SWN%20Conway%20Office~an.'
f = 'C:/temp/DS_Html_Email.html'
t = 'C:/temp/DS_Text_Email.txt'
dsUpdated = str(time.strftime('%A, %m/%d/%Y, %I:%M %p', time.localtime(os.path.getmtime(ds))))
KillObject(f)
KillObject(t)
WriteOutput(f,'\n\n

super-cooper » gis

Python and GIS – Beyond ModelBuilder and PythonWin

I’m published in ArcUser!

Mapping point locations with Python and Microsoft Live Search Maps

AAT Drill Schedule Mapper for Friday, 11/23/2007