20 Mbps data streaming behind PF

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
I'm still working on porting our unicast stream servers behind PF
firewall runs on  openbsd3.8.
That is a hidden, bridged firewall.

Now, Im testing with only one server, which is streaming from tcp port
443 to nearly 500-1000   instannt browser embedded java  clients.
Traffic makes up to 2 ~ 3Mbps now, but in real envriron.  it wil take
up to 20Mbps

At begginning, there was high numerical differences among  the OS's
ESTABLISHED netstat table, # of states in PF and # of connected hosts
of out server application, wihch is stable for years.
and my state table overwhelmed.

I customized PF rules as follows. And Differences has become

FURTHERMORE, I have noticed some unexpected BLOCKs at  pflogd0
Because the traffic, I have targetted to pass over PF is 10 times
bigger than the situation at now;
I'm cautions about those BLOCKED PACKETS.

There are 2 sample symthoms below ,

symptom #1:
rule 10/(match) block in on fxp0: >server1.443: F
0:0(0) ack 1 win 8621
rule 10/(match) block in on fxp0: > server1.443: P
22321829:22321963(134) ack 1951655540 win 16744
 rule 10/(match) block in on fxp0: > server1.443: R
2749563789:2749563789(0) win 64631

esp. This host one more connection on state table
 rule 10/(match) block in on fxp0: > server1.443: F
2282441566:2282441566(0) ack 646237210 win 65020
and 4 more blocks with ( F 0:0(0) ack 1 win 65020)

symptom #2:
 rule 11/(match) block out on fxp0: server1.443 >
[|tcp] (DF)

# Now I  block 1  ~ 5 connections  per minute  with avg  2 conn pr min
( I'm not sure those have SYN flag set. I hope those have not;) and
dont know how to investigate

Especially the first one!
I explaind the secound one , BLOCK OUT,  with the time limits i have
custmized on tcp options of the rule (  tcp.established 300)
because my box has already killed the state,because of unactivity  OS
behind FW tries to sending data.
-- because my server streams realtime quatos, 5 min inactivity is
unusal. ---

BUT, i have sone doubts about the first  one related to INBOUND BLOCKS
 -- sometimes those hosts ( blocked)  has already one or more states in
my table  ( but it is verry common , more than one client sharing the
same LAN connect to our servers). But blocked source ports are
different from the ones in state table, - ofcourse -
 -- some of the hosts  could never establish a connection yet, I have
checked that those are all acknowledged. Now, they are trying :) but
has not complaint yet:)
-- none of the limits have exceeded.  (max, max-src-states,
max-src-conn ..etc)

What may  cause these problems ?
   -- QUEUE limits ?
   --  TCP options ( S/SA) ?
   -- or what can else , which I have missed ?

Thank you for your help and forgive my this long msg takes time.

App. Developer

#pftop -vqueue
QUEUE                         BW SCH  PRIO     PKTS    BYTES   DROP_P
  any_eurojava                   14M cbq     2  2644516  558110K
6     1019    0  45521  11386    1314  403213

Some of My pf.conf

set block-policy drop
set state-policy floating

#set optimization aggressive
set optimization normal

set limit states 100000

set timeout { tcp.first 120,    tcp.opening 30, tcp.established 3600}
set timeout { tcp.closing 300,  tcp.finwait 45, tcp.closed 60 }
set timeout { udp.first 60,     udp.single 30,  udp.multiple 60 }
set timeout { icmp.first 20,    icmp.error 10 }
set timeout { other.first 60,   other.single 30, other.multiple 60 }
set timeout { adaptive.start 10000, adaptive.end 30000 }

set loginterface $ext_if

set skip on lo0
set skip on $int_if

scrub in on $ext_if all no-df
scrub    on $ext_if all reassemble tcp

################# QUEUEING CLASSES ################
altq on  $ext_if cbq bandwidth  20Mb queue {

 # Main Internet services of <secure> servers -
queue internet  bandwidth 16Mb
# EuroJava Service for any customers
        queue any_eurojava bandwidth 14Mb priority 2 qlimit  500
cbq(borrow ecn)
        queue any_web      bandwidth 1Mb priority 4 cbq(borrow ecn)
        queue any_commons  bandwidth 1Mb priority 4 cbq(default borrow
queue firewall          bandwidth 1Mb   priority 1 cbq(borrow ecn) #
firewall management queue
# secure servers, behind FW , established out queue
queue secure_out        bandwidth 1Mb   priority 5 cbq(borrow ecn)
# trusted hosts' traffic  on un-common services
queue trusted_in        bandwidth 1Mb   priority 3 cbq(borrow ecn)
# throttled access for flooding hosts on throttled queuee
queue throttled         bandwidth 1Mb   priority 7 cbq(red)

# Rule 10
block in  log  on $ext_if all

# Rule 11
block out log  on $ext_if all

block in log quick on $ext_if from <bad> to  any


# my main PASS rule for tcp in to port 443

pass  in on $ext_if proto tcp from any to <eurojava_servers>  port
  flags S/SA \
        tag CHECK_MORE keep state ( max 5000, max-src-states 100, \
        tcp.established 300, tcp.closing 10, tcp.finwait 10, tcp.closed
10  \
        max-src-conn 100, overload <throttle>) \
     queue any_eurojava
# CHECK rule for flooting customers
pass  in on $ext_if from <throttle> tagged CHECK_MORE tag THROTTLED
keep state  \
         ( max 500,  tcp.established 60,  tcp.closing 10, tcp.closed 5,
tcp.finwait 10 \
                max-src-conn-rate 50/5, overload <bad> flush global)
        queue throttled

pass out on $ext_if proto tcp from <secure> to any  flags S/SA modulate
state queue secure_out

Re: 20 Mbps data streaming behind PF

# tcpdump -n -e -o -vvv  -ttt -i pflog0 port 443

rule 10/(match) [uid 0, pid 1807] block in on fxp0:
Quoted text here. Click to load it

block in on fxp0: > server1.443: [|tcp] (ttl 250, id
62897, len 40, bad cksum 0! differs by 7430

rule 11/(match) [uid 0, pid 1807] block out on fxp0: server1.443 > [|tcp] (DF) (ttl 64, id 20448, len 40)

Re: 20 Mbps data streaming behind PF

thanks Rmkml

I also started to debug with clasicall method , checking out the rule
options which i have put with great enthusiasm to do the job as it must
done :(

queue is must, bandwith is also. scrube didnot change anything ,
is not for inbound rules
But i had to disable S/SA flags  :(

problem may be solved, i dont distinguish who is comming in any more :)

as the nature of PF,  S/SA is  right thing to do. but in man pages , I
noticed it may break some connections for specific os plaoform and
connection types.

like sne hosts send Tcp RST  for begginning of comminication.

The problem is seemed to be solved now thank yo

when i enable S/SA,  i get  blocks with  "bad cksum 0! " ??

I do not filtering anything on $int_if
But if i disable S/SA for internal -> out
then i would  get wrong  state tables from in -> out and wrong queue
will be assinged and my state table will be overwhelmed again. ( esp at
times FW reset)

now, I am sure I know the reson of outbound blocks,  those are all
inactive connections, and killed from state table  due to time limit,
not important.it must be, OS netstat may  increase but it is also not
imp. because of established timeout of OS.

if i can find what TCP flags I should put in place of S/SA as tcp
option to describe the comminucation beginning, i will be verry happy.

----- Original Message -----
Sent: Monday, April 10, 2006 1:42 PM
Subject: Re: 20 Mbps data streaming behind PF

Quoted text here. Click to load it

Site Timeline