MS Exchange NLB Multicast mode with HP Procurve switches

Flood NLB ExchangeВесь текст с точки зрения сетевого специалиста, а не администратора Microsoft Exchange и серверов Windows. Здесь все, что касается сути проблемы и ее решения на сетевом уровне.

Проблема

Кластер Exchange включает в себя три сервера, которые для избыточности используют NLB в режиме unicast. Режим прекрасен тем, что сетевая карта каждого из трех серверов  отвечает на один и тот же MAC-адрес, который присвоен NLB кластеру. Как результат — все свитчи в LAN сходят с ума, т.к. они не в состоянии заполнить свою CAM-table и вынуждены флудить весь трафик до кластера через все свои порты (кроме того, на который пакет пришел). И это ужасно. В нашем случае вышло что каждый хост в сети постоянно ловит около 10Мбит/сек юникастового флуда, который этому хосту не предназначен, а который интересен только серверу Exchange. У нас даже есть несколько IP-телефонов, подключенных к 10Мбит/сек интерфейсам, которые сразу почувствовали неладное, т.к. для них «места в проводе» не осталось — это, кстати, и заставило копать что не так.

Решение

Наименее трудоемких/затратных решений — два.

Первое — засунуть весь NLB/Exchange кластер в отдельный VLAN, после чего трафик до кластера и от него при взаимодействии с основной LAN будет маршрутизироваться и флуд в LAN самоустранится. Решение нормальное, но нужно менять IP-адреса серверов кластера, что в сети большой организации с несколькими сотнями постоянно работающих с почтой сотрудников сделать крайне сложно.

Второе, которое нам больше подходит — переключить кластер на работу в режиме NLB multicast. Что это означает? Это означает то, что каждый сервер кластера на своей сетевой карте будет иметь уникальный юникастовый адрес (IP и MAC), но кроме этого, у каждого из серверов будет настроен общий IP с мультикастовым MAC адресом, вида 0100:5exx:xxxx. Что этот MAC означает для коммутатора? Это зависит от того, активирован ли на коммутаторе протокол IGMP или нет.

Если IGMP выключен/не поддерживается — проблема не решится никак. Будет такой же флуд, как и был при работе с NLB Unicast. Как коммутатор работает с обычными юникастовыми MAC-адресами? Например, ты делаешь ping 192.168.0.1. Хост, с которого запущен ping, шлет ARP-запрос, спрашивая, на ком висит этот IP адрес. ARP запрос имеет MAC адрес назначения ffff:ffff:ffff и  каждый коммутатор пересылеат его во все порты. Некий хост в сети, который владеет IP 192.168.0.1 отвечает, что это он и что MAC у него 5df4:0033:34f2. Коммутатор видит, что ответ прилетел на порт A24 и теперь он в своей таблице CAM (таблица соответствия порт — mac адрес) делает запись 5df4:0033:34f2 —> А24. Теперь, когда кто-то захочет послать трафик на 5df4:0033:34f2, коммутатор просто перешлет фрейм в порт A24 а не во все дырки.

Что будет если некто с IP 192.168.0.1 ответит, что у него MAC адрес — 0100:5e7f:04f9 (мультикаст)? В этом случае он не сделает никакой записи в своей CAM-table и снова и снова все фреймы, адресованные на 0100:5e7f:04f9 ,будет слать во все порты. Свитч всегда будет игнорировать фреймы с мультикастовыми source MAC адресами с позиции занесения их в CAM-table. Так, как правило, работает ОС  коммутатора.

Что будет если включить на коммутаторе IGMP? Ну тогда все станет гораздо лучше. Свитч, с активированным IGMP теперь внимательно наблюдает за пролетающими через него IGMP-запросами/ответами. NLB кластер в режиме Multicast IGMP будет слать запросы IGMP Join, говорящие о том, что он (кластер, каждый его сервер) хочет получать фреймы, адресованные на 0100:5exx:xxxx (мультикаст адрес кластера). Видя подобные запросы любой коммутатор с  IGMP теперь начинает слать трафик, адресованный 0100:5exx:xxxx только в те порты, на которых он поймал IGMP Join от серверов кластера. Если на коммутаторе включен IGMP, но ни одного IGMP Join он не увидел, фреймы на 0100:5exx:xxxx будут отбрасываться как ненужные, т.к. никто явно не заявил о том, что хочет их получать. И это радует. Проблема тут может возникнуть в том случае,  в сети есть коммутаторы, которые в принципе не знают что такое IGMP. Тогда нужно извращаться, например, по первому предложенному варианту, или использовать внешний баллансировщик вместо NLB, но это уже совсем другая история.

Итак, как и зачем нужен IGMP для оптимизации NLB понятно, осталось реализовать. Как это делается со стороны кластера Microsoft Windows я не видел, мне не интересно, но мануалов по этой теме хватает. Что нужно делать на коммутаторах — в тексте ниже.

В нашем случае вся коммутируемая сеть — это свитчи HP ProCurve. Два коммутатора уровня ядра — 5412zl, и около 80 коммутаторов уровня распределения/доступа моделей 2910al, 2810 и пр. Все поддерживают IGMP.

Активировать IGMP на каждом коммутаторе невероятно сложно.  Команда (у нас IPGM активировался для VLAN 1, т.к. в нем располжен кластер NLB/Exchange):

switch_x(config)# vlan 1 ip igmp

Все. Нужно ввести эту команду на каждом коммутаторе  и после этого можно считать, что IGMP в LAN активирован и почти готов. «Почти» потому, что в этой конструкции пока не хватает того, кто называется IGMP Querier. IGMP Querier — это маршрутизатор или L3 Коммутатор, выполняющий некую активную роль в работе IGMP. Он слушает IGMP Join запросы и периодически опрашивает тех, от кого прилетал IGMP Join, по прежнему ли они нуждаются в траффике, который запрашивают. IGMP Querier в пределах одной подсети (VLAN) всегда один. Настроено их может быть несколько, но только один будет активным. Остальные коммутаторы, не являющиеся Querier, пассивно слушают IGMP Join/Query сообщения и, на их основе, принимают решение, что делать с мультикастовыми фреймами.

В нашем случае избранным стал один из коммутаторов ядра HP ProCurve 5412zl. Настройка выглядит так:

switch_x(config)#vlan 1 ip igmp querier

Теперь севреры кластера NLB и коммутатор уровня ядра успешно обмениваются IGMP Query/Join сообщениями, а все остальные коммутаторы пассивно слушают это общение и на основе услышанного пересылают или не пересылают мультикаст фреймы в нужную сторону.

В нашем случае кластер NLB имеет IP адрес 10.200.4.249. Посмотрим, что видно на одном из коммутаторов ядра:

Core1# sh ip igmp groups

IGMP Group Address Information

VLAN ID Group Address   Expires       UpTime        Last Reporter   | Type
——- ————— ————- ————- ————— + ——
1       224.0.1.24      0h 2m 23s     3d 1h 37m 16s 10.200.4.51     | Filter
1       224.0.1.41      0h 2m 23s     3d 1h 37m 16s 10.200.4.120    | Filter
1       239.192.211.136 0h 2m 21s     3d 1h 37m 16s 10.200.4.152    | Filter
1       239.255.4.249   0h 2m 18s     2d 14h 50m 2s 10.200.4.249    | Filter
1       239.255.17.18   0h 2m 15s     3d 1h 37m 21s 10.200.16.56    | Filter
1       239.255.219.45  0h 2m 17s     3d 1h 37m 17s 10.200.4.119    | Filter
………………………………….//………………………………………………………………

Тут видно, что коммутатор создал для IP кластера мультикастовую группу   239.255.4.249.

Смотрим дальше:

Core1# sh ip igmp

IGMP Service Protocol Info

Total VLANs with IGMP enabled                : 1
Current count of multicast groups joined     : 16

IGMP Filter Unknown Multicast: Disabled
IGMP Filter Unknown Multicast Status: Disabled

VLAN ID : 1
VLAN Name : DEFAULT_VLAN
IGMP version : 2
Querier Address [this switch] : 10.200.0.1
Querier Port :
Querier UpTime : 3d 1h 39m 24s
Querier Expiration Time : 0h 0m 5s

Active Group Addresses Type       Expires         Ports      Reports Queries
———————- ———- ————— ———- ——- ——-
224.0.1.24             Filter     0h 2m 24s       E3,J1,Trk1 6315    0
224.0.1.41             Filter     0h 2m 19s       K10        2106    0
239.192.211.136        Filter     0h 4m 19s       H1-H3      6321    0
239.255.4.249          Filter     0h 4m 17s       J1         1798    0

……………………………………//……………………………………………………….

Тут видно, что именно этот коммутатор является IGMP Querier и видно, что фреймы для мультикастовой группы 239.255.4.249, которая соответствет IP адресу кластера 10.200.4.249 данный коммутатор будет слать только в порт J1.

Посмотрим что видно на любом другом коммутаторе:

 

Other_Switch# sh ip igmp

Status and Counters — IP Multicast (IGMP) Status

VLAN ID : 1
VLAN Name : DEFAULT_VLAN
Querier Address : 10.200.0.1

Active Group Addresses Reports Queries Querier Access Port
———————- ——- ——- ——————-
224.0.1.60             6639    1995    Trk1
224.168.168.168        5988    1995    Trk1
232.44.44.233          108     106     Trk1
239.255.255.250        26549   2318    Trk1

Видно, что коммутатор знает, кто для него является IGMP Querier (тот самый коммутатор уровня ядра). При этом, вывод команды не отображает мультикастовую группу кластера NLB (может есть другая команда, но я ее не нашел). Однако, так как флуд в сети после внесения обозначенных настроек и включения режима мультикаст на серверах NLB пропал совсем, можно сделать вывод, что все работает. Коммутаторы шлют трафик на адрес кластера только через те интерфейсы, которые в сторону кластера смотрят и через которые пролетали IGMP Join сообщения.

Вроде бы все, но есть еще один нюанс. Коммутатор ядра помимо прочего является еще и маршрутизатором, соединяющим различные VLANы сети. И, если для хостов, находящихся в той же подсети что и кластер Exchange никаких проблем не возникло, то клиенты, сидящие в других подсетях после внесения всех настроек не смогли достучаться до почтового сервера. Почему? Тут снова нужно представить что происходит. Например, есть клиент с IP 10.201.0.100, который находится в VLAN 2, а не VLAN 1, где находится кластер с IP 10.200.4.249. Клиент шлет некий запрос на почтовый сервер. Для этого он использует свой default gateway с IP 10.201.0.1 (IP адрес VLAN 2 SVI на коммутаторе ядра). Коммутатор ядра (уже как маршрутизатор) должен переслать этот пакет на адрес 10.200.4.249. Но чтобы это сделать он должен узнать его MAC? Но тут то и возникает проблема. Дело в том, что свитч в данном случае работает так, что не принимает ответы на свои ARP запросы, если в них возвращается мультикастовый IP. Т.е. если на своем ПК в сети 10.200.0.0 я могу сделать arp -a 10.200.4.249 и увидеть:

адрес в Интернете      Физический адрес      Тип
10.200.4.249          01-00-5e-7f-04-f9     динамический

То свитч так сделать не может. Он не покажет этот физический адрес, т.к. не примет его в качестве ответа на свой ARP. Решене оказалось простым. Нужно сказать свитчу, чтобы он, не смотря на свою принципиальность, все таки принимал в ARP ответах мультикастовый MAC. Команда выглядит так:

Core1# ip arp-mcast-replies

Вот после этого все начинает работать как часы и никакого флуда. Минус 10 Мбит/сек. паразитного трафика:)


Не забывайте оставлять комментарии, если пост был вам полезен!
Опубликовано в Сети
0 comments on “MS Exchange NLB Multicast mode with HP Procurve switches
258 Пинги/Обратные ссылки для "MS Exchange NLB Multicast mode with HP Procurve switches"
  1. Google

    We came across a cool site that you could possibly enjoy. Take a appear if you want.

  2. online direct lender payday loans

    […]Wonderful story, reckoned we could combine some unrelated data, nevertheless truly really worth taking a search, whoa did one study about Mid East has got a lot more problerms as well […]

  3. bondage restraints

    […]usually posts some extremely exciting stuff like this. If you are new to this site[…]

  4. fifty shades of grey toys

    […]very couple of websites that transpire to be comprehensive below, from our point of view are undoubtedly well worth checking out[…]

  5. fantasy whip

    […]very few internet websites that occur to be in depth beneath, from our point of view are undoubtedly properly worth checking out[…]

  6. bondage toys kit

    […]below you’ll uncover the link to some web pages that we believe you’ll want to visit[…]

  7. best bondage toys

    […]usually posts some pretty interesting stuff like this. If you’re new to this site[…]

  8. master series vibrating dildo

    […]Here is a great Weblog You may Find Intriguing that we Encourage You[…]

  9. боли при половом акте

    […]very handful of internet websites that transpire to become comprehensive beneath, from our point of view are undoubtedly well worth checking out[…]

  10. rabbit ears toy

    […]the time to study or take a look at the material or web sites we’ve linked to below the[…]

  11. ангина сколько длится

    […]always a major fan of linking to bloggers that I appreciate but really don’t get a whole lot of link love from[…]

  12. pravil’noe pitanie na mesjac

    […]Every as soon as inside a though we pick out blogs that we study. Listed beneath are the most recent web-sites that we pick out […]

  13. True Cinnamon Sri Lanka

    […]Here are some of the websites we recommend for our visitors[…]

  14. instamp3

    […]here are some links to internet sites that we link to since we think they are worth visiting[…]

  15. 토토사이트

    […]check below, are some absolutely unrelated sites to ours, however, they are most trustworthy sources that we use[…]

  16. شركة نقل عفش الفروانية

    […]usually posts some quite interesting stuff like this. If you’re new to this site[…]

  17. 사다리사이트

    […]very few web sites that come about to be in depth below, from our point of view are undoubtedly properly really worth checking out[…]

  18. 먹튀검증

    […]Wonderful story, reckoned we could combine several unrelated data, nonetheless truly really worth taking a search, whoa did one particular understand about Mid East has got much more problerms as well […]

  19. 안전놀이터

    […]Sites of interest we’ve a link to[…]

  20. valjutnye uslovija

    […]Sites of interest we have a link to[…]

  21. silicone sex toys

    […]usually posts some incredibly fascinating stuff like this. If you’re new to this site[…]

  22. best vibrator

    […]we came across a cool website which you may possibly enjoy. Take a appear in case you want[…]

  23. immagini del pensiero

    […]one of our guests lately proposed the following website[…]

  24. two headed dildo

    […]the time to study or take a look at the content or internet sites we’ve linked to below the[…]

  25. сайт

    […]we like to honor quite a few other online web sites around the web, even when they aren’t linked to us, by linking to them. Below are some webpages worth checking out[…]

  26. sex toys for beginners adam and eve

    […]the time to study or check out the content or sites we have linked to beneath the[…]

  27. beast iptv

    […]one of our visitors just lately encouraged the following website[…]

  28. online virtual service

    […]Here is a superb Weblog You may Obtain Interesting that we Encourage You[…]

  29. Affiliate Marketing Platform

    […]The details talked about in the report are a few of the very best out there […]

  30. best buy israel

    […]check beneath, are some completely unrelated websites to ours, however, they are most trustworthy sources that we use[…]

  31. double head dildo

    […]Here is a good Weblog You might Obtain Fascinating that we Encourage You[…]

  32. секс знакомства

    […]Here is a good Weblog You may Come across Interesting that we Encourage You[…]

  33. 안전공원 추천

    […]check beneath, are some entirely unrelated internet websites to ours, nonetheless, they are most trustworthy sources that we use[…]

  34. Pay someone to take my online class

    […]we prefer to honor lots of other internet web pages on the web, even if they aren’t linked to us, by linking to them. Under are some webpages worth checking out[…]

  35. Bursa Saham Indonesia

    […]just beneath, are quite a few totally not related sites to ours, however, they may be certainly really worth going over[…]

  36. Raw Food Chef Training

    […]Here is a superb Weblog You may Discover Interesting that we Encourage You[…]

  37. Saham Reksadana Indonesia

    […]very handful of internet websites that transpire to become comprehensive beneath, from our point of view are undoubtedly properly really worth checking out[…]

  38. 토토사이트

    […]below you will locate the link to some web sites that we feel you ought to visit[…]

  39. curso design grafico online

    […]Here is an excellent Weblog You may Come across Interesting that we Encourage You[…]

  40. professional IT staffing services orlando

    […]we prefer to honor a lot of other world-wide-web sites around the web, even though they aren’t linked to us, by linking to them. Underneath are some webpages really worth checking out[…]

  41. xnnx:

    xnnx

    […]The facts mentioned in the post are some of the best readily available […]

  42. fbv cursos gratuitos

    […]below you will uncover the link to some websites that we feel you must visit[…]

  43. Criminal Defense Lawyer Roanoke, VA

    […]we prefer to honor many other web web-sites on the internet, even when they aren’t linked to us, by linking to them. Beneath are some webpages worth checking out[…]

  44. Nail Salon Scottsdale

    […]just beneath, are various entirely not associated internet sites to ours, however, they may be certainly worth going over[…]

  45. male stroker

    […]one of our visitors lately recommended the following website[…]

  46. vibrator for women

    […]that could be the end of this article. Right here you’ll come across some web-sites that we believe you will enjoy, just click the links over[…]

  47. fretboard

    […]always a major fan of linking to bloggers that I like but don’t get quite a bit of link love from[…]

  48. PCD Pharma Franchise Companies In Telangana

    […]very few web-sites that occur to be in depth below, from our point of view are undoubtedly well really worth checking out[…]

  49. Best penis pumps

    […]we came across a cool site that you could possibly love. Take a search for those who want[…]

  50. Super Mario subscription box

    […]Sites of interest we have a link to[…]

  51. Rabbit vibrator

    […]very handful of websites that take place to be comprehensive below, from our point of view are undoubtedly properly really worth checking out[…]

  52. How to use g spot vibrator

    […]check beneath, are some completely unrelated web sites to ours, even so, they may be most trustworthy sources that we use[…]

  53. Clitoral vibrator review

    […]Here are several of the websites we advocate for our visitors[…]

  54. Couples vibrators review

    […]please go to the web-sites we stick to, like this one, because it represents our picks in the web[…]

  55. DSA Apuntes Apunte

    […]below you will uncover the link to some web pages that we consider you ought to visit[…]

  56. apunts 8443

    […]Every when inside a although we select blogs that we study. Listed below are the most recent sites that we select […]

  57. Memoria P2 — DSBM Práctica

    […]just beneath, are many completely not associated web sites to ours, however, they’re surely really worth going over[…]

  58. https://judislotterpercaya.blogspot.com/p/ingin-tau-cara-paling-mudah-menang.html

    […]that may be the end of this report. Here you’ll locate some sites that we believe you will enjoy, just click the links over[…]

  59. wild cats

    […]just beneath, are quite a few totally not connected web pages to ours, even so, they’re certainly really worth going over[…]

  60. bondage gear

    […]check below, are some totally unrelated web-sites to ours, having said that, they may be most trustworthy sources that we use[…]

  61. Summary — Book «The Art Of Public Speaking» — 16 Chapter Exam Review Material

    […]although web-sites we backlink to below are considerably not connected to ours, we feel they are really worth a go by means of, so have a look[…]

  62. Vouchers for shopping

    […]The information talked about inside the write-up are a few of the top readily available […]

  63. kegel ball exercises

    […]although internet sites we backlink to beneath are considerably not connected to ours, we really feel they are essentially worth a go by means of, so have a look[…]

  64. Dard shayari

    […]we like to honor several other world wide web sites on the internet, even when they aren’t linked to us, by linking to them. Under are some webpages worth checking out[…]

  65. silicone rabbit

    […]please check out the websites we follow, such as this one, because it represents our picks through the web[…]

  66. FIFA 19 Coins

    […]Wonderful story, reckoned we could combine several unrelated data, nevertheless truly worth taking a appear, whoa did one study about Mid East has got additional problerms also […]

  67. luxe vibrator

    […]the time to read or visit the subject material or web pages we’ve linked to beneath the[…]

  68. mini wand massager

    […]please go to the internet sites we follow, including this a single, as it represents our picks from the web[…]

  69. how to play with vibrato

    […]below you’ll come across the link to some sites that we feel you should visit[…]

  70. g spot vibrator review

    […]check beneath, are some completely unrelated internet sites to ours, however, they may be most trustworthy sources that we use[…]

  71. bullet vibe

    […]always a large fan of linking to bloggers that I like but don’t get a good deal of link like from[…]

  72. evolved sex toy

    […]Wonderful story, reckoned we could combine a few unrelated data, nevertheless actually really worth taking a look, whoa did 1 master about Mid East has got far more problerms also […]

  73. realistic suction cup dildo

    […]we like to honor numerous other net sites around the net, even when they aren’t linked to us, by linking to them. Under are some webpages worth checking out[…]

  74. gay anal sex

    […]below you will locate the link to some sites that we consider it is best to visit[…]

  75. remote control vibrator

    […]Every when inside a when we decide on blogs that we study. Listed beneath would be the most up-to-date web-sites that we opt for […]

  76. vibrating prostate massager

    […]Wonderful story, reckoned we could combine a few unrelated information, nonetheless genuinely really worth taking a appear, whoa did one learn about Mid East has got a lot more problerms at the same time […]

  77. Check Here

    […]the time to read or pay a visit to the subject material or internet sites we’ve linked to below the[…]

  78. leak detection repair

    […]just beneath, are numerous entirely not related web sites to ours, on the other hand, they’re certainly really worth going over[…]

  79. data entry work from home

    […]just beneath, are many completely not related internet sites to ours, nonetheless, they’re surely really worth going over[…]

  80. grand rapids seo

    […]that is the end of this post. Right here you’ll uncover some web pages that we feel you’ll value, just click the links over[…]

  81. what house can i afford

    […]The info talked about inside the report are several of the very best offered […]

  82. Google

    That may be the finish of this article. Right here you’ll locate some sites that we think you will value, just click the links.

  83. top anal sex toys

    […]the time to study or check out the content material or web pages we’ve linked to below the[…]

  84. Expensive

    […]although web-sites we backlink to below are considerably not related to ours, we feel they are essentially worth a go via, so possess a look[…]

  85. best male enhancement pumps

    […]check below, are some entirely unrelated internet websites to ours, however, they’re most trustworthy sources that we use[…]

  86. adam and eve vibrating kegel balls

    […]one of our guests just lately proposed the following website[…]

  87. strapless strap on

    […]Wonderful story, reckoned we could combine a number of unrelated data, nonetheless truly really worth taking a search, whoa did a single learn about Mid East has got much more problerms as well […]

  88. best anal plug reviews

    […]Here is a great Weblog You may Discover Exciting that we Encourage You[…]

  89. fun88

    […]we came across a cool site that you simply could get pleasure from. Take a appear for those who want[…]

  90. 188bet

    […]always a huge fan of linking to bloggers that I appreciate but really don’t get a good deal of link really like from[…]

  91. apps for windows pc download

    […]very couple of internet sites that come about to be in depth below, from our point of view are undoubtedly effectively worth checking out[…]

  92. free download for windows 7

    […]Every as soon as in a whilst we choose blogs that we study. Listed below are the most up-to-date internet sites that we pick […]

  93. download pc games for windows

    […]Wonderful story, reckoned we could combine some unrelated data, nonetheless genuinely really worth taking a search, whoa did 1 study about Mid East has got much more problerms too […]

  94. pc games for windows 10

    […]usually posts some really fascinating stuff like this. If you’re new to this site[…]

  95. pc games download

    […]please pay a visit to the websites we comply with, like this one particular, as it represents our picks from the web[…]

  96. pc games for windows 10

    […]here are some links to web-sites that we link to mainly because we believe they’re really worth visiting[…]

  97. free download for pc windows

    […]Every the moment in a while we pick blogs that we read. Listed below would be the most up-to-date web pages that we pick […]

  98. pc games for windows 7

    […]we came across a cool web site that you simply may well appreciate. Take a appear should you want[…]

  99. app download for pc

    […]the time to read or take a look at the content material or web pages we’ve linked to below the[…]

  100. free download for pc windows

    […]that may be the finish of this article. Right here you’ll obtain some internet sites that we think you’ll enjoy, just click the links over[…]

  101. best male enhancement pumps

    […]we like to honor quite a few other web web sites around the net, even when they aren’t linked to us, by linking to them. Under are some webpages really worth checking out[…]

  102. Google

    We came across a cool internet site that you just may enjoy. Take a appear for those who want.

  103. free apps for pc download

    […]Here is a great Weblog You might Find Interesting that we Encourage You[…]

  104. free games download for pc

    […]Sites of interest we’ve a link to[…]

  105. apps download for windows 7

    […]although websites we backlink to beneath are considerably not associated to ours, we feel they are actually worth a go by way of, so possess a look[…]

  106. героический самоотверженный поступок

    […]the time to read or pay a visit to the content material or internet sites we have linked to beneath the[…]

  107. демократичный и демократический

    […]Here is an excellent Blog You might Come across Interesting that we Encourage You[…]

  108. катится любовь

    […]one of our guests a short while ago encouraged the following website[…]

  109. chevrolet dealer houston tx

    […]usually posts some incredibly interesting stuff like this. If you are new to this site[…]

  110. pc games download

    […]here are some hyperlinks to web sites that we link to mainly because we consider they may be really worth visiting[…]

  111. rencontres francais

    […]Every as soon as inside a even though we decide on blogs that we study. Listed below are the most recent web sites that we pick out […]

  112. cameras ao vivo

    […]check below, are some completely unrelated web sites to ours, even so, they may be most trustworthy sources that we use[…]

  113. rabbit toy

    […]Here is a great Blog You might Locate Exciting that we Encourage You[…]

  114. 3 Piece Anal Play Kit

    […]although internet websites we backlink to below are considerably not associated to ours, we really feel they may be really really worth a go as a result of, so possess a look[…]

  115. cleaning out for anal sex

    […]that could be the finish of this post. Here you will come across some internet sites that we think you’ll appreciate, just click the hyperlinks over[…]

  116. Zonnebrand

    […]one of our guests a short while ago proposed the following website[…]

  117. realistic male stroker

    […]Here are a few of the web sites we recommend for our visitors[…]

  118. Speelgoedwatersproeier

    […]just beneath, are several entirely not connected internet sites to ours, on the other hand, they are certainly worth going over[…]

  119. https://www.bazaraki.com/

    […]always a major fan of linking to bloggers that I love but do not get quite a bit of link adore from[…]

  120. app download for pc

    […]the time to read or take a look at the subject material or web sites we’ve linked to below the[…]

  121. apps download for windows 10

    […]check below, are some completely unrelated internet sites to ours, on the other hand, they’re most trustworthy sources that we use[…]

  122. apps for windows pc download

    […]The info talked about inside the write-up are a number of the best readily available […]

  123. free pc games download

    […]please take a look at the websites we comply with, which includes this 1, because it represents our picks through the web[…]

  124. pc games for windows 8

    […]very few web sites that transpire to be comprehensive beneath, from our point of view are undoubtedly effectively worth checking out[…]

  125. app download for windows

    […]Wonderful story, reckoned we could combine some unrelated data, nonetheless definitely really worth taking a search, whoa did a single discover about Mid East has got far more problerms as well […]

  126. free download for windows 10

    […]we came across a cool web-site that you simply could get pleasure from. Take a appear when you want[…]

  127. apps for pc

    […]Here are some of the websites we recommend for our visitors[…]

  128. pc games free download

    […]below you will uncover the link to some websites that we assume it is best to visit[…]

  129. bullet massager

    […]we like to honor quite a few other web web pages on the net, even if they aren’t linked to us, by linking to them. Under are some webpages really worth checking out[…]

  130. top couples toy

    […]please stop by the web pages we follow, including this a single, as it represents our picks through the web[…]

  131. apps download for windows 10

    […]the time to study or take a look at the material or web-sites we’ve linked to beneath the[…]

  132. Gurgaon

    […]we prefer to honor quite a few other world-wide-web sites around the net, even though they aren’t linked to us, by linking to them. Beneath are some webpages worth checking out[…]

  133. cursos gratuitos

    […]we prefer to honor a lot of other net internet sites around the internet, even if they aren’t linked to us, by linking to them. Below are some webpages worth checking out[…]

  134. ring gay

    […]Here is an excellent Blog You may Come across Fascinating that we Encourage You[…]

  135. dani pussy

    […]one of our guests not too long ago advised the following website[…]

  136. bullet vibrator review

    […]Wonderful story, reckoned we could combine several unrelated data, nevertheless genuinely worth taking a appear, whoa did one particular discover about Mid East has got much more problerms also […]

  137. vibrating wand massager

    […]although web-sites we backlink to below are considerably not associated to ours, we really feel they may be in fact worth a go by means of, so possess a look[…]

  138. new rabbit vibrator

    […]The facts mentioned within the write-up are several of the very best out there […]

  139. Bàn làm việc đứng tự động

    […]Sites of interest we have a link to[…]

  140. webcam babes

    […]that will be the finish of this report. Right here you will come across some web sites that we believe you’ll appreciate, just click the hyperlinks over[…]

  141. sexe webcam

    […]Here are a few of the web-sites we recommend for our visitors[…]

  142. how to use blowyo extreme wave

    […]just beneath, are various absolutely not connected web-sites to ours, nonetheless, they are surely worth going over[…]

  143. prostate toy for men

    […]one of our visitors a short while ago proposed the following website[…]

  144. thrusting massager

    […]very couple of websites that transpire to become in depth beneath, from our point of view are undoubtedly nicely really worth checking out[…]

  145. anal butt plug

    […]usually posts some pretty fascinating stuff like this. If you are new to this site[…]

  146. green card organization

    […]Here are a number of the sites we recommend for our visitors[…]

  147. doc johnson ultra skin

    […]just beneath, are quite a few entirely not associated sites to ours, having said that, they’re surely worth going over[…]

  148. powerful silicone vibrator

    […]below you’ll obtain the link to some web-sites that we believe you must visit[…]

  149. anal vibrator

    […]usually posts some really intriguing stuff like this. If you’re new to this site[…]

  150. самые популярные фамилии в мире

    […]Here are several of the internet sites we advise for our visitors[…]

  151. https://www.movieschor.org/genre/hindi-dubbed-hollywood-movie-south-indian-movies/

    […]please pay a visit to the websites we follow, such as this a single, because it represents our picks in the web[…]

  152. слушать песню ланфрен ланфра

    […]we came across a cool internet site that you just could possibly appreciate. Take a appear in case you want[…]

  153. омонимы словарь

    […]just beneath, are a lot of entirely not associated web sites to ours, nevertheless, they are certainly really worth going over[…]

  154. sex toy store

    […]check beneath, are some absolutely unrelated websites to ours, even so, they are most trustworthy sources that we use[…]

  155. https://www.bloglovin.com/@oneworldherald/cyber-crime-with-mac-users

    […]although internet sites we backlink to beneath are considerably not related to ours, we feel they’re in fact worth a go via, so have a look[…]

  156. tiger tyson cock

    […]that will be the end of this post. Right here you will come across some websites that we think you’ll value, just click the hyperlinks over[…]

  157. pocket stroker

    […]always a significant fan of linking to bloggers that I enjoy but do not get lots of link love from[…]

  158. strap on harness for pegging

    […]we came across a cool web-site that you simply may possibly take pleasure in. Take a look if you want[…]

  159. how to increase penis size naturally

    […]The info mentioned in the report are some of the best available […]

  160. ΝΤΕΤΕΚΤΙΒ ΘΕΣΣΑΛΟΝΙΚΗ

    […]that would be the finish of this post. Right here you will discover some web sites that we feel you’ll appreciate, just click the hyperlinks over[…]

  161. vibrator sex toy

    […]always a significant fan of linking to bloggers that I like but don’t get lots of link like from[…]

  162. best vibrating panty

    […]below you’ll obtain the link to some internet sites that we feel you ought to visit[…]

  163. anal trainer kit

    […]Every as soon as in a even though we pick out blogs that we read. Listed below are the newest web sites that we select […]

  164. Pornland

    […]Here are some of the sites we advise for our visitors[…]

  165. Kamagra Gold

    […]Sites of interest we have a link to[…]

  166. quick loans bad credit

    […]although web sites we backlink to beneath are considerably not associated to ours, we feel they’re in fact worth a go via, so have a look[…]

  167. megapu.sh

    […]always a big fan of linking to bloggers that I like but do not get quite a bit of link love from[…]

  168. jelly caribbean

    […]Wonderful story, reckoned we could combine a number of unrelated information, nonetheless truly worth taking a search, whoa did one master about Mid East has got more problerms also […]

  169. scrap car removal chilliwack

    […]Here are several of the web-sites we advocate for our visitors[…]

  170. android app tester

    […]The facts mentioned inside the report are a number of the very best obtainable […]

  171. songtext down under

    […]the time to study or stop by the material or web pages we have linked to beneath the[…]

  172. текст песни сид и нэнси

    […]one of our visitors just lately advised the following website[…]

  173. eminem kim перевод

    […]usually posts some really exciting stuff like this. If you’re new to this site[…]

  174. enlargement pumps results

    […]please check out the web pages we stick to, which includes this one particular, because it represents our picks through the web[…]

  175. silicone anal plug

    […]check beneath, are some completely unrelated websites to ours, nevertheless, they’re most trustworthy sources that we use[…]

  176. best dildos

    […]below you will find the link to some web pages that we assume it is best to visit[…]

  177. ben wa balls

    […]that would be the finish of this post. Here you will uncover some websites that we feel you will enjoy, just click the hyperlinks over[…]

  178. king dildo

    […]here are some links to internet sites that we link to because we feel they are really worth visiting[…]

  179. interview questions on induction motor pdf

    […]Wonderful story, reckoned we could combine some unrelated data, nonetheless genuinely worth taking a look, whoa did a single learn about Mid East has got more problerms too […]

  180. anal sex in water

    […]just beneath, are quite a few absolutely not connected web-sites to ours, even so, they are surely really worth going over[…]

  181. monogamy game for couples

    […]usually posts some very intriguing stuff like this. If you’re new to this site[…]

  182. fantasy panties

    […]that may be the finish of this post. Right here you’ll discover some sites that we feel you will value, just click the hyperlinks over[…]

  183. 15% minoxidil

    […]Wonderful story, reckoned we could combine some unrelated information, nevertheless seriously really worth taking a appear, whoa did a single understand about Mid East has got far more problerms at the same time […]

  184. cock ring during sex

    […]Every after inside a though we pick out blogs that we study. Listed below would be the most recent internet sites that we pick […]

  185. Anabolic Steroids

    […]very couple of sites that transpire to be in depth beneath, from our point of view are undoubtedly well really worth checking out[…]

  186. cryptocurrency influencer

    […]the time to read or go to the content or sites we have linked to beneath the[…]

  187. satta king

    […]Sites of interest we have a link to[…]

  188. bullet and egg vibrator

    […]Sites of interest we have a link to[…]

  189. red hot blaze vibrator

    […]please stop by the websites we stick to, which includes this one particular, as it represents our picks from the web[…]

  190. best dual ended dildo

    […]we came across a cool website that you just could possibly get pleasure from. Take a appear for those who want[…]

  191. best pocket pussy

    […]here are some hyperlinks to sites that we link to for the reason that we assume they are really worth visiting[…]

  192. how to use dildo

    […]Every when in a whilst we select blogs that we read. Listed below are the latest websites that we opt for […]

  193. website design west palm beach

    […]that will be the finish of this write-up. Right here you will find some internet sites that we think you will appreciate, just click the links over[…]

  194. bullet vibrator review

    […]Sites of interest we’ve a link to[…]

  195. dilator review

    […]the time to read or check out the content material or internet sites we have linked to below the[…]

  196. sucking masturbator

    […]The info mentioned within the write-up are a few of the top offered […]

  197. nipple clamps

    […]here are some hyperlinks to websites that we link to simply because we believe they’re worth visiting[…]

  198. porn:

    porn

    […]check below, are some totally unrelated web-sites to ours, having said that, they are most trustworthy sources that we use[…]

  199. pornland

    […]check below, are some completely unrelated internet websites to ours, nonetheless, they’re most trustworthy sources that we use[…]

  200. waterproof vibrator

    […]The data mentioned within the post are several of the best offered […]

  201. penis vibrator

    […]Here are some of the sites we recommend for our visitors[…]

  202. minoxidil with azelaic acid

    […]Here is a good Blog You might Discover Interesting that we Encourage You[…]

  203. jockstrap bulge

    […]The facts mentioned within the post are several of the top out there […]

  204. oxford dictionary translate english to afrikaans

    […]Here is a great Weblog You might Obtain Exciting that we Encourage You[…]

  205. masturbator for him

    […]Sites of interest we’ve a link to[…]

  206. online dating sites

    […]the time to study or stop by the material or web pages we have linked to below the[…]

  207. realistic male stroker

    […]Here are a number of the web sites we suggest for our visitors[…]

  208. http://www.skipbinhireau.com

    […]we came across a cool web site which you could enjoy. Take a appear in case you want[…]

  209. commission

    […]Here are a few of the web pages we advise for our visitors[…]

  210. ssru.ac.th

    […]Sites of interest we’ve a link to[…]

  211. giant dildo

    […]although web-sites we backlink to beneath are considerably not related to ours, we really feel they may be actually really worth a go via, so have a look[…]

  212. bunny tickler

    […]although sites we backlink to beneath are considerably not associated to ours, we feel they may be basically worth a go by, so have a look[…]

  213. sex toys for him

    […]Wonderful story, reckoned we could combine a number of unrelated information, nevertheless actually really worth taking a appear, whoa did one particular study about Mid East has got extra problerms as well […]

  214. big black dildo

    […]here are some links to web-sites that we link to mainly because we feel they are worth visiting[…]

  215. powerful bullet massager

    […]that would be the end of this write-up. Right here you’ll discover some web pages that we think you’ll value, just click the links over[…]

  216. paddle sextoy

    […]always a large fan of linking to bloggers that I adore but really don’t get a whole lot of link like from[…]

  217. newest vibrator

    […]always a major fan of linking to bloggers that I adore but really don’t get a great deal of link really like from[…]

  218. top vibrating panties

    […]the time to study or stop by the material or web pages we’ve linked to beneath the[…]

  219. bullet vibrator unboxing

    […]usually posts some incredibly intriguing stuff like this. If you’re new to this site[…]

  220. bondage for beginners

    […]The information and facts mentioned in the write-up are a few of the best obtainable […]

  221. Goedkoopste cocktailprikkers

    […]always a huge fan of linking to bloggers that I adore but really don’t get quite a bit of link love from[…]

  222. Christine Reidhead, Educator

    […]Here is a great Blog You might Locate Fascinating that we Encourage You[…]

  223. kingcock dildo

    […]always a huge fan of linking to bloggers that I like but really don’t get a lot of link love from[…]

  224. 10% minoxidil with azelaic acid

    […]Every once in a whilst we choose blogs that we study. Listed below are the most current web-sites that we choose […]

  225. Hundetæppe

    […]here are some links to web sites that we link to since we assume they are worth visiting[…]

  226. Julia Ann

    […]Every when inside a whilst we opt for blogs that we study. Listed beneath would be the most current web pages that we choose […]

  227. https://se.mut.ac.ke/contact.html

    […]very couple of web-sites that happen to become in depth below, from our point of view are undoubtedly effectively worth checking out[…]

  228. bdjobstotal

    […]we like to honor several other world-wide-web internet sites around the internet, even when they aren’t linked to us, by linking to them. Underneath are some webpages worth checking out[…]

  229. Зарплата

    […]here are some links to web-sites that we link to mainly because we believe they’re really worth visiting[…]

  230. Intuit quickbooks download

    […]check below, are some completely unrelated internet websites to ours, having said that, they are most trustworthy sources that we use[…]

  231. Fashion Station

    […]we came across a cool internet site which you may possibly get pleasure from. Take a look if you want[…]

  232. Pool Glass Tile

    […]Every after inside a even though we pick out blogs that we study. Listed beneath are the most current web-sites that we opt for […]

  233. секс видео

    […]here are some hyperlinks to web sites that we link to simply because we feel they are worth visiting[…]

  234. sexboer

    […]Every as soon as in a though we choose blogs that we study. Listed beneath would be the most current web-sites that we choose […]

  235. geile sex filmpjes

    […]that is the finish of this report. Here you’ll uncover some web sites that we consider you will enjoy, just click the hyperlinks over[…]

  236. gratis films

    […]that may be the finish of this report. Right here you will find some web pages that we believe you will value, just click the links over[…]

  237. pornofilmpjes

    […]Every when inside a although we choose blogs that we study. Listed below would be the most up-to-date web sites that we pick out […]

  238. waterbury ct plumbers

    […]always a large fan of linking to bloggers that I really like but do not get lots of link love from[…]

  239. liteblue login

    […]The information and facts talked about inside the article are a few of the ideal obtainable […]

  240. iptv 7+

    […]Every the moment inside a when we decide on blogs that we study. Listed below are the most recent internet sites that we decide on […]

  241. liteblue login

    […]just beneath, are many completely not associated internet sites to ours, however, they are certainly worth going over[…]

  242. diy dildo

    […]we prefer to honor several other internet web pages on the web, even if they aren’t linked to us, by linking to them. Under are some webpages really worth checking out[…]

  243. multi speed vibrator

    […]Wonderful story, reckoned we could combine a handful of unrelated data, nonetheless genuinely worth taking a search, whoa did 1 learn about Mid East has got much more problerms too […]

  244. 7 inches dildo

    […]very few web-sites that transpire to be detailed beneath, from our point of view are undoubtedly properly worth checking out[…]

  245. finger sex

    […]Every when in a even though we pick blogs that we study. Listed beneath would be the most current web-sites that we decide on […]

  246. калифорнийское течение

    […]The data talked about in the report are a number of the best available […]

  247. silicone vibrator

    […]The information and facts talked about in the article are several of the very best readily available […]

  248. возвышение у алтаря

    […]just beneath, are quite a few totally not associated internet sites to ours, however, they’re surely worth going over[…]

  249. визбор юрий

    […]The information and facts mentioned inside the post are several of the top obtainable […]

  250. russian translation hello

    […]although web sites we backlink to below are considerably not connected to ours, we feel they may be truly worth a go through, so possess a look[…]

  251. cock pump

    […]Sites of interest we’ve a link to[…]

  252. hundeseng til stor hund

    […]always a large fan of linking to bloggers that I appreciate but really don’t get quite a bit of link love from[…]

  253. nipple clips

    […]the time to read or check out the content or internet sites we have linked to beneath the[…]

  254. bombshell dildo

    […]The facts talked about in the write-up are several of the ideal accessible […]

  255. massage in maadi

    […]Wonderful story, reckoned we could combine a few unrelated information, nevertheless genuinely really worth taking a search, whoa did a single discover about Mid East has got much more problerms also […]

  256. https://www.myassociatelogin.org/

    […]Here is a good Weblog You might Obtain Interesting that we Encourage You[…]

  257. Bluechew reviews

    […]Here are a number of the web sites we advise for our visitors[…]

Заодно посмотрите мои фоты в моем профиле вконтакте. Любые вопросы по существу статей можете задать там же.
Hostenko — лучший WordPress-хостинг