[NCLUG] Networking Issue

Tue Aug 6 10:38:59 MDT 2013

On Tue, 6 Aug 2013, Matt Kassawara wrote:

> I suspect "Tx discards" reflect buffer limitations somewhere... which
> explains why enabling flow control end-to-end mostly solves the discards
> problem but negatively impacts throughput.

That's what I was suspecting and am trying to get a downtime on the switch 
to change the buffering setup.

>
> I'm not familiar with HP's mesh technology, but if it amounts to link
> aggregation without equal-cost load balancing, you may see discards due to
> issues with the hashing algorithm.

All I really know is it's intended for performance, load-balancing and 
redundancy.  It's supposed to select the best network path between 2 end 
points on the network and it's supposed to update this over time based on 
utilization, etc...  From HP's white paper on meshing with regard to path 
selection.

  * Outbound queue depth, or the current outbound load factor, for any given
    outbound port in a possible path
  * Port speed, based on factors such as 10 Mbps, 100 Mbps, 1000 Mbps (or
    1 Gbps), 10 Gbps, full-duplex, or half-duplex
  * Inbound queue depth for any destination switch in a possible path
  * Increased packet drops, indicating an overloaded port or switch

>
> For 1 Gbps and faster connectivity, I would strongly consider implementing
> end-to-end jumbo frames to maximize performance.  Depending on the buffer
> architecture, it could positively or negatively impact the discards
> problem.  On modern equipment, I've only seen improvements from
> implementing it.

We have looked a little at going to Jumbo frames.  We are a heavy NFS user 
and > 50% of our traffic is metadata (getattr, etc..).   How does jubmo 
frames play into the metadata?

>
> P.S. - Isn't the 2500 mostly an ancient 10/100 Mbps switch with limited
> support for 1 Gbps?

The 2400 is a 10/100.  The 2500 is all gig ports with a set of 4 for 
creating a 4G trunked uplink.   No buffering to speak of..

>
>
> On Tue, Aug 6, 2013 at 9:06 AM, Michael Coffman
> <coffman at ftc.avagotech.com>wrote:
>
>> On Mon, 5 Aug 2013, Matt Kassawara wrote:
>>
>>  Do both variants of configuration A have the same problem?
>>>
>>
>> Yes.
>>
>>
>>  Do any of the switches in question indicate the reason for discards?
>>>
>>
>> Only the hp5412(b) listed is showing discards.   All it says from what I
>> can tell is 'Tx discards'
>>
>>
>>  Did you configure end-to-end support for jumbo frames?
>>>
>> Jumbo frames are not enabled.
>>
>>
>>  Can you provide more details on "10g mesh" between the 5412s?
>>>
>>
>> We have 3 5412's that are all connected via 10g and have HP's meshing
>> enabled.   What kind of detail are you looking for?
>>
>>
>>>
>>> On Mon, Aug 5, 2013 at 5:14 PM, Michael Coffman
>>> <coffman at ftc.avagotech.com>**wrote:
>>>
>>>
>>>> I don't do networking and our corporate networking group has been cut so
>>>> thin, they don't do much networking anymore either :(
>>>>
>>>> This is not a linux problem, but I know there are folks on this list that
>>>> have deep networking experience.  I'll try to describe the problem as
>>>> best
>>>> as I can in ascii.
>>>>
>>>> *NA = NetApp
>>>> *CS = Compute Server
>>>>
>>>> #----  A
>>>> NA -10g-> hp5412(a) -10g mesh-> hp5412(b) -1g-> CS
>>>> NA -10g-> hp5412(a) -10g mesh-> hp5412(b) -4g trunk-> hp2500 -1g-> CS
>>>>
>>>> o The read path to the compute server is very slow 20-30MB/s
>>>> o The write path to the NetApp is close to wire speed 110-130MB/s
>>>> o The hp5412(b) switch is having thousands of discards at the port to the
>>>>   CS system
>>>> o Enabling flow control on all the switches on the mesh almost
>>>>   completely eliminates the discards and increases read performance, but
>>>>   hurts write performance
>>>> o Reversing the flow above does not have the same performance problems
>>>> o Reversing the flow above does not cause discards on hp5412(a)
>>>>
>>>> #---- B
>>>> NA -10g-> hp5412(a) -10g mesh-> hp5412(b) -10g-> hp5800 -1g-> CS
>>>>
>>>> o The above scenario has no performance issues.
>>>> o Still seeing discardson hp5412(b) due to other traffic on the switch
>>>>
>>>> #---- C
>>>> NA -10g-> hp5412(a) -1g-> CS
>>>> NA -10g-> hp5412(a) -1g-> hp2500 -1g-> CS
>>>>
>>>> o The above scenario works fine.
>>>> o The hp5412(a) is not seeing the discards that hp5412(b) does.
>>>>
>>>> Any ideas on how to go about trouble shooting this?
>>>>
>>>> I am not looking for free answers to this problem here, I would be glad
>>>> to
>>>> talk to someone about doing some network consulting.  Also any pointers
>>>> to
>>>> a networking consultant/guru would be greatly appreciated.
>>>>
>>>> Thanks.
>>>> -MichaelC
>>>> ______________________________****_________________
>>>>
>>>> NCLUG mailing list       NCLUG at lists.nclug.org
>>>>
>>>> To unsubscribe, subscribe, or modify
>>>> your settings, go to:
>>>> http://lists.nclug.org/****mailman/listinfo/nclug<http://lists.nclug.org/**mailman/listinfo/nclug>
>>>> <http://**lists.nclug.org/mailman/**listinfo/nclug<http://lists.nclug.org/mailman/listinfo/nclug>
>>>>>
>>>>
>>>>  ______________________________**_________________
>>> NCLUG mailing list       NCLUG at lists.nclug.org
>>>
>>> To unsubscribe, subscribe, or modify
>>> your settings, go to:
>>> http://lists.nclug.org/**mailman/listinfo/nclug<http://lists.nclug.org/mailman/listinfo/nclug>
>>>
>>>
>> -MichaelC
>> ______________________________**_________________
>> NCLUG mailing list       NCLUG at lists.nclug.org
>>
>> To unsubscribe, subscribe, or modify
>> your settings, go to:
>> http://lists.nclug.org/**mailman/listinfo/nclug<http://lists.nclug.org/mailman/listinfo/nclug>
>>
> _______________________________________________
> NCLUG mailing list       NCLUG at lists.nclug.org
>
> To unsubscribe, subscribe, or modify
> your settings, go to:
> http://lists.nclug.org/mailman/listinfo/nclug
>

-MichaelC