instance get more than one fixed ip(grizzly-g3) - Ask OpenStack: Q&A Site for OpenStack Users and Developers
Description: i have install grizzly-g3, but quantum does not work well, when i boot 128 instances, i found one of instances got more than one fixed ip, howerver, when i boot 64 intances, it nerver happened, besides that , sometimes i can not ping vm with floatingip, i did not find any error message in my quantum log( all the files in the /var/log/quantum), follows are the error output and configurations | 97a93600-38e2-4700-9851-15ef56c1d628 | slave | ACTIVE | demo-int-net=172.16.100.4 | | 99aeb6b8-4252-4839-a7d1-f87853116100 | slave | ACTIVE | demo-int-net=172.16.100.117 | | 9aa82a35-c9f1-4f44-a108-d14e74eec231 | slave | ACTIVE | demo-int-net=172.16.100.108, 172.16.100.109 | | 9b6b1289-c450-4614-b647-e5ebdffff80a | slave | ACTIVE | demo-int-net=172.16.100.5 | | 9e0d3aa5-0f15-4b24-944a-6d6c3e18ce64 | slave | ACTIVE | demo-int-net=172.16.100.35 | | 9ea62124-9128-43cc-acdd-142f1e7743d6 | slave | ACTIVE | demo-int-net=172.16.100.132 | my setup : one db host(db service), one glance host(glance service) [cell cut: 3773 characters, first 1000 kept]
Keywords: ,Ask, OpenStack,forum,community,cloud,iaas

First time here? Check out the FAQ!

English ▼
Hi there! Please sign in tags users badges help

| ALL UNANSWERED |  | Ask Your Question

1

# instance get more than one fixed ip(grizzly-g3)

- neutron
- migrated

asked 2013-03-26 01:09:38 -0600

xinxin-shu
11 ●1 ●2 ●3

i have install grizzly-g3, but quantum does not work well, when i boot 128 instances, i found one of instances got more than one fixed ip, howerver, when i boot 64 intances, it nerver happened, besides that , sometimes i can not ping vm with floatingip, i did not find any error message in my quantum log( all the files in the /var/log/quantum), follows are the error output and configurations

| 97a93600-38e2-4700-9851-15ef56c1d628 | slave | ACTIVE | demo-int-net=172.16.100.4 | | 99aeb6b8-4252-4839-a7d1-f87853116100 | slave | ACTIVE | demo-int-net=172.16.100.117 | | 9aa82a35-c9f1-4f44-a108-d14e74eec231 | slave | ACTIVE | demo-int-net=172.16.100.108, 172.16.100.109 | | 9b6b1289-c450-4614-b647-e5ebdffff80a | slave | ACTIVE | demo-int-net=172.16.100.5 | | 9e0d3aa5-0f15-4b24-944a-6d6c3e18ce64 | slave | ACTIVE | demo-int-net=172.16.100.35 | | 9ea62124-9128-43cc-acdd-142f1e7743d6 | slave | ACTIVE | demo-int-net=172.16.100.132 |

my setup : one db host(db service), one glance host(glance service), on api host(keystone,nova-api,nova-scheduler, nova-conductor, quantum-server,quantum-dhcp, quantum-l3-agent,quantum-plugin-openvswitch-agent), eight compute host(each host with nova-compute, quantum-plugin-openvswitch-agent), i check that all the service on all hosts works well

i used vlan type network and openvswitch plugin:

my quantum.conf

[DEFAULT]

# Default log level is INFO

# verbose and debug has the same result.

# One of them will set DEBUG log level output

debug = True

# Address to bind the API server

bind_host = 0.0.0.0

# Port the bind the API server to

bind_port = 9696

# Quantum plugin provider module

# core_plugin =

core_plugin = quantum.plugins.openvswitch.ovs_quantum_plugin.OVSQuantumPluginV2

# Advanced service modules

# service_plugins =

# Paste configuration file

api_paste_config = /etc/quantum/api-paste.ini

# The strategy to be used for auth.

# Supported values are 'keystone'(default), 'noauth'.

auth_strategy = keystone

# Modules of exceptions that are permitted to be recreated

# upon receiving exception data from an rpc call.

# allowed_rpc_exception_modules = quantum.openstack.common.exception, nova.exception

# AMQP exchange to connect to if using RabbitMQ or QPID

control_exchange = quantum

# RPC driver. DHCP agents needs it.

notification_driver = quantum.openstack.common.notifier.rpc_notifier

# default_notification_level is used to form actual topic name(s) or to set logging level

default_notification_level = INFO

# Defined in rpc_notifier, can be comma separated values.

# The actual topic names will be %s.%(default_notification_level)s

notification_topics = notifications

[QUOTAS]

# resource name(s) that are supported in quota features

# quota_items = network,subnet,port

# default number of resource allowed per tenant, minus for unlimited

# default_quota = -1

# number of networks allowed per tenant, and minus means unlimited

# quota_network = 10

# number of subnets allowed per tenant, and minus means unlimited

# quota_subnet = 10

# number of ports allowed per tenant, and minus means unlimited

quota_port = 5000 quota_floatingip = 5000

# default driver to use for quota checks

# quota_driver = quantum.quota.ConfDriver

# =========== items for agent management extension =============

# Seconds to regard the agent as down.

# agent_down_time = 5

# =========== end of items for agent management extension =====

[DEFAULT_SERVICETYPE]

# Description of the default service type (optional)

# description = "default service type"

# Enter a service definition line for each advanced service provided

# by the default service type.

# Each service definition should be in the following format:

# <service>:<plugin>[:driver]

[SECURITYGROUP]

# If set to true this allows quantum to receive proxied security group calls from nova

# proxy_mode = False

[AGENT] root_helper = sudo quantum-rootwrap /etc/quantum ... (more)

edit retag flag offensive close merge delete
add a comment

## 31 answers

Sort by » oldest newest most voted
« previous1...1 2 3 4...4 next »
0

answered 2013-03-27 22:36:13 -0600

snaiksat
1

Yes, I am waiting for this to show up again. We see this in the automated infrastructure. Unfortunately in our setup it's unpredictable, but has shown up fairly often in the past.

edit flag offensive delete link more
add a comment
0

answered 2013-03-27 21:04:16 -0600

danwent
1

Sumit, if you are able to see this predictably, can you insert some additional logging calls to understand the flow that leads to these duplicate ports? For example, are both calls made within the same call to allocate_for_instance (e.g., is the the nets array somehow containing the same network twice?). Or is this the result of multiple calls to allocate_for_instance? Logging a msg with the instance-id when we enter allocate_for_instance, then having a log message indicating the quantum port ids allocated for the instance-id would be valuable (I would argue that latter should be something we actually push to nova for inclusion, as it seems appropriate at the INFO or DEBUG level.

edit flag offensive delete link more
add a comment
0

answered 2013-03-27 20:42:58 -0600

snaiksat
1

We have seen this behavior about 20% of the time, not deterministic in our setup. We were running one instance of nova-compute on the host.

edit flag offensive delete link more
add a comment
0

answered 2013-03-27 19:27:36 -0600

arosen
96 ●3 ●6 http://blog.aaronorose...

Hi,

I've looked into this and tried to reproduce it but have been unsuccessful. I don't see anything off hand in the code either :/ . One thing I tried was running two instances of nova-compute on the same host which actually does work. When doing this I ran the following script several times (see if a vm gets two ips from this network....delete all vms...run again).. but didn't see the issue.

$ cat boot.sh nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf &

Just to rule it out is there any chance that you might have two instances of nova-compute running on the same host? Could you provide the following output?

ps -eaf | grep nova

and (just for the heck of it the output to this so we can test with the same type of network shared/provider etc.. in the rare change this matters) quantum net-show 8b933d20-f53b-40e0-a96d-67f3befb6449 quantum subnet-show 24be33c7-64f9-4695-811d-33eeb4ad75da

Thanks

edit flag offensive delete link more
add a comment
0

answered 2013-03-27 19:27:35 -0600

arosen
96 ●3 ●6 http://blog.aaronorose...

Hi,

I've looked into this and tried to reproduce it but have been unsuccessful. I don't see anything off hand in the code either :/ . One thing I tried was running two instances of nova-compute on the same host which actually does work. When doing this I ran the following script several times (see if a vm gets two ips from this network....delete all vms...run again).. but didn't see the issue.

$ cat boot.sh nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf & nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=effd0a91-923e-4345-a368-5235b3238bf6 asdf &

Just to rule it out is there any chance that you might have two instances of nova-compute running on the same host? Could you provide the following output?

ps -eaf | grep nova

and (just for the heck of it the output to this so we can test with the same type of network shared/provider etc.. in the rare change this matters) quantum net-show 8b933d20-f53b-40e0-a96d-67f3befb6449 quantum subnet-show 24be33c7-64f9-4695-811d-33eeb4ad75da

Thanks

edit flag offensive delete link more
add a comment
0

answered 2013-03-26 17:42:15 -0600

snaiksat
1

I will let Xinxin report his observations, in our case I did see two network interfaces and tap devices being created:

<interface type="bridge"> <mac address="fa:16:3e:a8:fe:96"/> <model type="virtio"/> <driver name="qemu"/> <source bridge="qbr7aac0341-19"/> <target dev="tap7aac0341-19"/> <filterref filter="nova-instance-instance-00000004-fa163ea8fe96"> <parameter name="IP" value="10.194.193.4"/> <parameter name="DHCPSERVER" value="10.194.193.2"/> <parameter name="PROJNET" value="10.194.193.0"/> <parameter name="PROJMASK" value="255.255.255.0"/> </filterref> </interface> <interface type="bridge"> <mac address="fa:16:3e:9a:b9:33"/> <model type="virtio"/> <driver name="qemu"/> <source bridge="qbr7ddce382-c7"/> <target dev="tap7ddce382-c7"/> <filterref filter="nova-instance-instance-00000004-fa163e9ab933"> <parameter name="IP" value="10.194.193.5"/> <parameter name="DHCPSERVER" value="10.194.193.2"/> <parameter name="PROJNET" value="10.194.193.0"/> <parameter name="PROJMASK" value="255.255.255.0"/> </filterref> </interface>

edit flag offensive delete link more
add a comment
0

answered 2013-03-26 17:35:02 -0600

salvatore-orlando
76 ●2

Hi Xinxin,

I do apologise, but I need to ask you more information in order to nail down this issue.

From your tests, does the total number of ports created equate the number of VMs? In other words, do you see any VM without ports, other than VMs with two ports?

I am also wondering if when Quantum creates two ports, nova also creates two VIFs. From the look of the port-show output, as both NICs are ACTIVE, this seems to be the case, but it's probably worth checking the generated libvirt XML file (or logging in into the instance and check for network interfaces)

Also, in order to understand which code paths are activated, how do you specify networks at boot? Do you use --nic network-id, --nic port-id, or you do not specify the --nic option at all?

edit flag offensive delete link more
add a comment
0

answered 2013-03-26 07:00:48 -0600

snaiksat
1

For whatever it's worth, we have seen this issue a number of times too in our tests and I believe this was happening even prior to grizzly-3. Seems to happen when a bunch of instances are created consecutively (like in a script).

edit flag offensive delete link more
add a comment
0

answered 2013-03-26 05:25:02 -0600

xinxin-shu
11 ●1 ●2 ●3

sorry about that i click on the wrong button, i clean all nova-compute.log on compute nodes, and reproduce this error output, there are two instances with multiple fixed ip,

fb356afb-49ce-4922-82c4-c8460cb8fc4c | slave | ACTIVE | demo-int-net=172.16.100.129 | | fbf44f20-27b0-4c13-8630-8fafeecff06c | slave | ACTIVE | demo-int-net=172.16.100.24 | | fd3f892e-c94a-407e-9dab-edb1c48adafc | slave | ACTIVE | demo-int-net=172.16.100.117, 172.16.100.115 | | ff04d946-1851-4050-8d02-65f4fffb42de | slave | ACTIVE | demo-int-net=172.16.100.50 | 55321d5f-9af1-4736-9599-412742f792ae | slave | ACTIVE | demo-int-net=172.16.100.64 | | 555c5198-8a44-4dee-98eb-ef704b324f8b | slave | ACTIVE | demo-int-net=172.16.100.111, 172.16.100.112 | | 557dd69b-ff73-4d67-bf48-6f860ba3b236 | slave | ACTIVE | demo-int-net=172.16.100.63 |

the compute log of instance 555c5198-8a44-4dee-98eb-ef704b324f8b is http://paste.openstack.org/show/34592/

+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+ | 21833908-28f0-4ec7-9497-d19eba06e209 | | fa:16:3e:e0:b3:cc | {"subnet_id": "24be33c7-64f9-4695-811d-33eeb4ad75da", "ip_address": "172.16.100.111"} | | 9637e6e5-673e-4443-9b0e-27f15ed3f92b | | fa:16:3e:bb:f0:f4 | {"subnet_id": "24be33c7-64f9-4695-811d-33eeb4ad75da", "ip_address": "172.16.100.112"} | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+

quantum port-show

root@kapi:/var/log/quantum# quantum port-show 21833908-28f0-4ec7-9497-d19eba06e209 +----------------------+---------------------------------------------------------------------------------------+ | Field | Value | +----------------------+---------------------------------------------------------------------------------------+ | admin_state_up | True | | binding:capabilities | {"port_filter": true} | | binding:vif_type | ovs | | device_id | 555c5198-8a44-4dee-98eb-ef704b324f8b | | device_owner | compute:None | | fixed_ips | {"subnet_id": "24be33c7-64f9-4695-811d-33eeb4ad75da", "ip_address": "172.16.100.111"} | | id | 21833908-28f0-4ec7-9497-d19eba06e209 | | mac_address | fa:16:3e:e0:b3:cc | | name | | | network_id | 8b933d20-f53b-40e0-a96d-67f3befb6449 | | security_groups | 065a1cba-03f8-4172-bcee-d635155b965a | | status | ACTIVE | | tenant_id | ebacbc9c99f84607920b2ac749608623 | +----------------------+---------------------------------------------------------------------------------------+

+----------------------+---------------------------------------------------------------------------------------+ | Field | Value | +----------------------+---------------------------------------------------------------------------------------+ | admin_state_up | True | | binding:capabilities | {"port_filter": true} | | binding:vif_type | ovs | | device_id | 555c5198-8a44-4dee-98eb-ef704b324f8b | | device_owner | compute:None | | fixed_ips | {"subnet_id": "24be33c7-64f9-4695-811d-33eeb4ad75da", "ip_address": "172.16.100.112"} | | id | 9637e6e5-673e-4443-9b0e-27f15ed3f92b | | mac_address | fa:16:3e:bb:f0:f4 | | name | | | network_id | 8b933d20-f53b-40e0-a96d-67f3befb6449 | | security_groups | 065a1cba-03f8-4172-bcee-d635155b965a | | status | ACTIVE | | tenant_id | ebacbc9c99f84607920b2ac749608623 | +----------------------+---------------------------------------------------------------------------------------+

from above ouput, the two ports are with same device_id. the other instance is same that two ports with same device id

edit flag offensive delete link more
add a comment
0

answered 2013-03-26 05:17:56 -0600

xinxin-shu
11 ●1 ●2 ●3

i clean all nova-compute.log on compute nodes, and reproduce this error output, there are two instances with multiple fixed ip,

fb356afb-49ce-4922-82c4-c8460cb8fc4c | slave | ACTIVE | demo-int-net=172.16.100.129 | | fbf44f20-27b0-4c13-8630-8fafeecff06c | slave | ACTIVE | demo-int-net=172.16.100.24 | | fd3f892e-c94a-407e-9dab-edb1c48adafc | slave | ACTIVE | demo-int-net=172.16.100.117, 172.16.100.115 | | ff04d946-1851-4050-8d02-65f4fffb42de | slave | ACTIVE | demo-int-net=172.16.100.50 | 55321d5f-9af1-4736-9599-412742f792ae | slave | ACTIVE | demo-int-net=172.16.100.64 | | 555c5198-8a44-4dee-98eb-ef704b324f8b | slave | ACTIVE | demo-int-net=172.16.100.111, 172.16.100.112 | | 557dd69b-ff73-4d67-bf48-6f860ba3b236 | slave | ACTIVE | demo-int-net=172.16.100.63 |

the compute log of instance 555c5198-8a44-4dee-98eb-ef704b324f8b is http://paste.openstack.org/show/34592/

+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+ | 21833908-28f0-4ec7-9497-d19eba06e209 | | fa:16:3e:e0:b3:cc | {"subnet_id": "24be33c7-64f9-4695-811d-33eeb4ad75da", "ip_address": "172.16.100.111"} | | 9637e6e5-673e-4443-9b0e-27f15ed3f92b | | fa:16:3e:bb:f0:f4 | {"subnet_id": "24be33c7-64f9-4695-811d-33eeb4ad75da", "ip_address": "172.16.100.112"} | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+

quan

edit flag offensive delete link more
add a comment
« previous1...1 2 3 4...4 next »

## Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

[hide preview]

# Get to know Ask OpenStack

Resources for moderators

## Question Tools

1 follower

## Stats

Asked: 2013-03-26 01:09:38 -0600

Seen: 232 times

Last updated: Jul 03 '13

## Related questions

A question about ovs-vlan-test

can't ping floating ip

about l3 agent

noVNC support for Google Chrome 14+

[mitaka][octavia] octavia-worker cannot reach amphora [closed]

what's the _junk in codes?

unable to assign floating ip to instance

Connecting OVS interfaces to a bond with VLAN tagging

neutron router iptables not hit

Howto Multiserver Stats

( 2016-11-08 11:32:09 -0600 )edit
none