using cloudbridge to access openstack on Jetstream

This notebook demostrates the use of the cloudbridge api to add files to buckes, list the available vm images, create networks and routers and how to create an instance. (The last step is not working completely. we will fix that one later.)

If you do not have cloudbridge installed uncomment the next line and run it.

In [54]:
#!pip install cloudbridge --upgrade
In [1]:
from cloudbridge.cloud.factory import CloudProviderFactory, ProviderList

you will want to log into the open stack portal for jetspeed to get you project name.

In [10]:
js_config = {"os_username": "your user name",
             "os_password": "and password",
             "os_auth_url": "https://jblb.jetstream-cloud.org:35357/v3",
             "os_user_domain_name": "tacc",
             "os_project_domain_name": "tacc",
             "os_project_name": "your project (see the portal)"}
js = CloudProviderFactory().create_provider(ProviderList.OPENSTACK, js_config)

we now have a handle to jetspeed! let's look at a list of the public vm images.

In [11]:
js.compute.images.list()
Out[11]:
[<CB-OpenStackMachineImage: ChromoSnapShot_b370dc51-2c95-496d-b4aa-19b52860863c_2017-01-30_09:44:41 (3153ba66-d60c-4591-859a-4786b96ff614)>,
 <CB-OpenStackMachineImage: R with Intel compilers (CentOS 7) (c112d8aa-31a1-4535-9c86-cd0decc3b1ce)>,
 <CB-OpenStackMachineImage: Myers L533 1st Image (dc7ce197-8217-4ff2-8e88-843a8094e427)>,
 <CB-OpenStackMachineImage: Myers L533 1st Image (f7a04d77-b267-4bef-93ba-b5b89b7f9e22)>,
 <CB-OpenStackMachineImage: Myers L533 1st Image (988a7951-3db1-4362-a29c-d19bcd65e89a)>,
 <CB-OpenStackMachineImage: Myers L533 1st Image (6b9faa14-a3dc-4e44-90dd-30dae0c33d93)>,
 <CB-OpenStackMachineImage: Intel Development (CentOS 7) (b5bfb513-1d79-49ff-aed0-b55c8fe0114b)>,
 <CB-OpenStackMachineImage: Chemcompute Slave Slurm (e4b7aa4e-1f13-4426-9b1b-152651114351)>,
 <CB-OpenStackMachineImage: Python3_Jupyter_Tensorflow (78e3a5c6-3aef-4030-b9b7-9899393643fe)>,
 <CB-OpenStackMachineImage: Centos 7 (7.2) Development GUI (fd3eafe8-7f07-4028-a87e-1b47ff558653)>,
 <CB-OpenStackMachineImage: Centos 7 (7.2) Development GUI (bbb79bdd-fd87-420f-bfee-727b3e0dc7b2)>,
 <CB-OpenStackMachineImage: MAKER 2.31.8 with CCTools (6102b30a-eab5-4589-92a8-8ab5917e1875)>,
 <CB-OpenStackMachineImage: MAKER 2.31.8 with CCTools (060883e7-ae7e-4a52-a296-b1cf0d877ed2)>,
 <CB-OpenStackMachineImage: Centos 7 (7.2) Development GUI (e852a0f1-110f-44a1-850a-de2c835f723f)>,
 <CB-OpenStackMachineImage: Brandeis_statistics_workshop (65fc486c-c6c3-472f-8387-54452844ba4d)>,
 <CB-OpenStackMachineImage: Centos 7 (7.3) Development GUI (4c25454d-8716-4cd5-841d-d00981536170)>,
 <CB-OpenStackMachineImage: Centos 7 (7.3) Development GUI (a76e3441-b5d0-43f9-bb10-35783b356bbb)>,
 <CB-OpenStackMachineImage: BioLinux 8 (906e09cf-2827-49c5-82fd-ea74aa527e1d)>,
 <CB-OpenStackMachineImage: R with Intel compilers (CentOS 7) (66322719-5bfa-4646-93c6-0132e9a958e5)>,
 <CB-OpenStackMachineImage: R with Intel compilers (CentOS 7) (53abecce-6b21-4d35-861d-e11528eafb32)>,
 <CB-OpenStackMachineImage: Centos 7 (7.2) Development GUI (39f013fc-015c-4f36-b784-d3e2254586a3)>,
 <CB-OpenStackMachineImage: Intel Development (CentOS 7) (e18aba82-c8c4-4eb7-8f46-bec610ad6c30)>,
 <CB-OpenStackMachineImage: R with Intel compilers (CentOS 7) (b25db736-d5e9-484d-a6ad-4431853a5c4c)>,
 <CB-OpenStackMachineImage: Ubuntu 14.04.3 Development GUI (21ffc12a-5cd7-4d8b-b3fc-2677cae72b8f)>,
 <CB-OpenStackMachineImage: Ubuntu 14.04.3 Development GUI (ce66671d-7d9f-4d38-9aef-e193ab21bca3)>,
 <CB-OpenStackMachineImage: Intel Development (CentOS 7) (1da1a8cc-311a-48b8-8418-aaba63668580)>,
 <CB-OpenStackMachineImage: MAKER v3 with CCTools (c643dab7-7ba6-4031-909f-8bff9e02b6af)>,
 <CB-OpenStackMachineImage: QGIS 2.18.1 (bcbe6b38-2190-48e6-9c50-e6c811c5f219)>,
 <CB-OpenStackMachineImage: Ubuntu 14.04 with Hadoop 2.7.3 (9e04c7e2-8527-4355-870a-ce825dbdcf8d)>,
 <CB-OpenStackMachineImage: MATLAB (CentOS 6) (f9dd55f2-c25b-4dcb-b95e-40d194965c78)>,
 <CB-OpenStackMachineImage: trusty-docker (a8df8653-c77a-4518-a4e9-2c59939fd0be)>,
 <CB-OpenStackMachineImage: I435-I535-B669 Project B (5ad04c2a-1503-4b44-9f5a-ad0abde2a685)>,
 <CB-OpenStackMachineImage: R with Intel compilers (CentOS 7) (428eaaab-d72d-45cd-b634-3ca4cd2f4101)>,
 <CB-OpenStackMachineImage: MATLAB (CentOS 6) (d6fe8a85-b43a-4d95-89b5-2e394bd57495)>,
 <CB-OpenStackMachineImage: Centos 7 (7.2) Development (93905450-0714-4336-a9c1-5e0f50aa9209)>,
 <CB-OpenStackMachineImage: Ubuntu 14.04.3 Development GUI (61f00773-be3f-45ad-b959-3ef2d7288f4c)>,
 <CB-OpenStackMachineImage: CentOS 6 (6.8) Development GUI (798830e0-9053-4700-a3bd-4acda9935cc2)>,
 <CB-OpenStackMachineImage: CentOS 6 (6.8) Development (d9f2c452-86e9-4327-a626-7fda2a50da4f)>,
 <CB-OpenStackMachineImage: XNIT Demo (cd2ac938-bcd4-4300-b33c-694034af2e8e)>,
 <CB-OpenStackMachineImage: project a (e8d017e0-f7ce-46b2-b141-81acbd10a30d)>,
 <CB-OpenStackMachineImage: project a (6c015e14-d6a1-40b4-85ca-ff3d62d12102)>,
 <CB-OpenStackMachineImage: I535-I435-B669 Project A (a8b24625-b8a8-4eb9-8ce5-b2b51a2a2aaf)>,
 <CB-OpenStackMachineImage: Galaxy 16.07 Standalone - 161021b01 (2cf07e4a-62a8-41c2-9282-f3c53962f296)>,
 <CB-OpenStackMachineImage: Ubuntu 16.04 LTS (87e08a17-eae2-4ce4-9051-c561d9a54bde)>,
 <CB-OpenStackMachineImage: I535-I435-B669 Project A (fcc8542f-1b06-4d51-80c9-c8a505bc6eff)>,
 <CB-OpenStackMachineImage: Galaxy in worm world (f56f4750-abfe-41e4-b61c-be662a5814ea)>,
 <CB-OpenStackMachineImage: Centos 7 (7.2) Development GUI (4c2f0709-00e5-49bc-bcbf-e452862854a1)>,
 <CB-OpenStackMachineImage: ubuntu-16.04-xenial-cloudimage-20160830 (a92d174e-375c-420c-bfcf-55101c448aa9)>,
 <CB-OpenStackMachineImage: CentOS 7 R Intel compilers (33c126bf-832c-47a4-ad14-b663fb8f7f10)>,
 <CB-OpenStackMachineImage: Intel Development (CentOS 7) (b4073d7d-d250-47b8-9060-28c4a0e74423)>]

the follow will list our current buckets and add a new one.

In [12]:
print js.object_store.list()
newbuck = js.object_store.create('my_new_bucket')
print(newbuck.name)
[<CB-OpenStackBucket: afourth_buck>, <CB-OpenStackBucket: escience>, <CB-OpenStackBucket: my_buck>, <CB-OpenStackBucket: my_buck2>, <CB-OpenStackBucket: my_bucket>, <CB-OpenStackBucket: my_new_buck>, <CB-OpenStackBucket: my_new_bucket>, <CB-OpenStackBucket: my_new_bucket2>, <CB-OpenStackBucket: my_new_bucket3>, <CB-OpenStackBucket: my_new_bucket4>]
my_new_bucket

let's get a handle to another new bucket

In [13]:
bucket = js.object_store.create('my_new_bucket2')
In [14]:
print bucket
<CB-OpenStackBucket: my_new_bucket2>

we can search for a bucket by name

In [15]:
buckes = js.object_store.find(name='my_new_bucket2')
In [16]:
buckes
Out[16]:
[<CB-OpenStackBucket: my_new_bucket2>]

Now lets's add some stuff to a blob we will call 'stuff'

In [17]:
buckobj = buckes[0].create_object('stuff')

now we upload a tiny file called 'stuf' it contains 2 lines

here is a file
of stuff
In [18]:
fo = open('c:\users\dennis\documents\stuff.txt','rb')
In [19]:
buckobj.upload(fo)
In [20]:
js.object_store.list()
Out[20]:
[<CB-OpenStackBucket: afourth_buck>,
 <CB-OpenStackBucket: escience>,
 <CB-OpenStackBucket: my_buck>,
 <CB-OpenStackBucket: my_buck2>,
 <CB-OpenStackBucket: my_bucket>,
 <CB-OpenStackBucket: my_new_buck>,
 <CB-OpenStackBucket: my_new_bucket>,
 <CB-OpenStackBucket: my_new_bucket2>,
 <CB-OpenStackBucket: my_new_bucket3>,
 <CB-OpenStackBucket: my_new_bucket4>]
In [21]:
buckes[0].list()
Out[21]:
[<CB-OpenStackBucketObject: stuff>]
In [22]:
stuff = buckes[0].get('stuff')
In [23]:
itr = stuff.iter_content()
In [24]:
for it in itr:
    print it
here is a file
of stuff

now let's see if we can start up an image. This has a bug so it doesn't work, but this shows what is supposed to work. We will update when the bug is fixed.

In [25]:
img = js.compute.images.get('9e04c7e2-8527-4355-870a-ce825dbdcf8d')
In [26]:
img
Out[26]:
<CB-OpenStackMachineImage: Ubuntu 14.04 with Hadoop 2.7.3 (9e04c7e2-8527-4355-870a-ce825dbdcf8d)>
In [27]:
inst_type = sorted([t for t in js.compute.instance_types.list()
                    if t.vcpus >= 4 and t.ram >= 8],
                   key=lambda x: x.vcpus*x.ram)[0]
In [28]:
inst_type
Out[28]:
<CB-OpenStackInstanceType: m1.medium (3)>

the following is how to create a key-pair. if it already exists you can grab it with the find() function.

In [64]:
#kp = js.security.key_pairs.create('cloudbridge_sec')
#with open('cloudbridge_sec.pem', 'w') as f:
#    f.write(kp.material)
#import os
#os.chmod('cloudbridge_sec.pem', 0400)
kp = js.security.key_pairs.find(name='cloudbridge_sec')
In [32]:
kp
Out[32]:
[<CBKeyPair: cloudbridge_sec>]
In [33]:
sg = js.security.security_groups.list()[0]
In [34]:
sg
Out[34]:
<CB-OpenStackSecurityGroup: 1eeae9e0-bf15-4089-9d23-9931625f8711>
In [43]:
netlist = js.network.list()
In [44]:
netlist
Out[44]:
[<CB-OpenStackNetwork: 4367cd20-722f-4dc2-97e8-90d98c25f12e (public)>,
 <CB-OpenStackNetwork: 0e04d98d-f8c6-4882-9e1a-fd321b0414da (genapp-net)>,
 <CB-OpenStackNetwork: 4d7cc5c5-1be2-4af3-a208-63a4cd538903 (scicloud)>,
 <CB-OpenStackNetwork: 1ae6a914-9003-4176-a2f0-57e1bb31c52e (scicloud2)>]
In [37]:
net = js.network.get('4d7cc5c5-1be2-4af3-a208-63a4cd538903')
In [38]:
net
Out[38]:
<CB-OpenStackNetwork: 4d7cc5c5-1be2-4af3-a208-63a4cd538903 (scicloud)>
In [42]:
net = js.network.create('scicloud2')
sn = net.create_subnet('10.0.0.0/28', 'scicloud2')
router = js.network.create_router('scicloud2')
In [61]:
if not net.external:
    for n in js.network.list():
        if n.external:
            external_net = n
            break
router.attach_network(external_net.id)
router.add_route(sn.id)

oops! the following doesn't work.

In [63]:
#inst = js.compute.instances.create( 
#    name='scicloud-VM', image=img, instance_type=inst_type,
#    key_pair=kp, security_groups=[sg], network=net)
In [ ]: