Discussion:
what is use OVERLAPPED,lpCompletionKey in GetQueuedCompletionStatus
(too old to reply)
sandeep
2006-05-12 15:07:05 UTC
Permalink
Hi,
I trying to write http proxy server(using tcp/ip) which can handle
10000 clients so I am trying to use CreateIoCompletionPort ,
GetQueuedCompletionStatus but I am not getting proper idea what is use
of OVERLAPPED,lpCompletionKey, I read about it in MSDN and The code
project but I didn't get it properly . so can any one explain it
with proper example
bluethought
2006-05-13 01:46:28 UTC
Permalink
CompletionKeys are DWORD values associated with your socket. When you
call CreateIoCompletionPort with a socket handle and CompletionKey,
that CompletionKey is associated with the socket handle, so that when
operations complete on that socket handle, the completion notifications
return that CompletionKey, to allow you to identify which socket it is.
I normally use as the CompletionKey the pointer to a per-socket
structure, cast to a DWORD. You can use indices to an array, or
anything else you like.

OVERLAPPED I/O is a topic in itself...an OverLapped structure is
required to post overlapped i/o requests that return completion
notifications on an IOCP (or to completion routines or by signalling
event objects if you choose to use that method). Overlapped I/O allows
you to post multiple I/O operations and wait for them to succeed.
OVERLAPPED structures are essentially Per-Io-Operation structures that
when returned from GetQueuedCompletionStatus also allow you to figure
out which operation succeeded. Theres much more to the topic of
OVERLAPPED structs.

Now read MSN and CodeProject and things should make more sense :) Also
take a look at Network Programming for MS Windows by Jones and Ohlund,
well worth the price (2nd ed preferably).

10000 clients might require a lot of resources because those will
entail the creation of 10000 client sockets (as opposed to sockets
returned from accept/AcceptEx). So you might want to scale down your
goals till you have the knowhow to measure your target system and
figure out whether your goals are achievable on that system (CPU,
Memory, OS).
Michael K. O'Neill
2006-05-13 17:43:54 UTC
Permalink
Post by bluethought
...
10000 clients might require a lot of resources because those will
entail the creation of 10000 client sockets (as opposed to sockets
returned from accept/AcceptEx). So you might want to scale down your
goals till you have the knowhow to measure your target system and
figure out whether your goals are achievable on that system (CPU,
Memory, OS).
Indeed. In fact, since it's a proxy server for 10,000 clients, it will need
20,000 sockets.
bluethought
2006-05-14 08:02:40 UTC
Permalink
Yes, and thats 10,000 sockets returned from accept/AcceptEx, and
another 10,000 you use with connect() or ConnectEx. The latter end up
using more resources than the former...on a win32 system you'll end up
running out of handles/Non Paged Pool because of that...while your
system might be able to handle more than 20,000 sockets of the kind
returned merely from accept/AcceptEx.
sandeep
2006-05-15 05:13:10 UTC
Permalink
Thank for all ,
I has written simple proxy server(tcp/ip) using multithreading and
select it is working perfect ,now I am trying to use
CreateIoCompletionPort , GetQueuedCompletionStatus but I am not getting
it so can any one send me http Simple proxy server(tcp/ip) code by
using CreateIoCompletionPort, GetQueuedCompletionStatus
sandeep
2006-05-15 15:12:22 UTC
Permalink
Problem in using CreateIoCompletionPort.

I am trying to write http proxy server but it waiting at
::GetQueuedCompletionStatus(g_main_port, &read_bytes,
(DWORD*)&key,&over, INFINITE); When I tried to connect through browser
.
I debug I found it is listening but not accepting because it is
waiting at (GQCS) so what is the mistake I did in using
CreateIoCompletionPort



unsigned int WINAPI ThreadProc(void *);

#define HTTP "http://"

HANDLE g_work_port;
HANDLE g_main_port;

typedef struct _session
{
SOCKET acceptfd;
}SESSION;

void main()
{

int clinu=0;
unsigned threadID;

//-------------------------------server ---------------------------
SOCKET acceptfd;
int server[150],nready,socketfd,nrec=0,nsend=0,listenfd=0,len;
char serverbuf[1024]={0};
struct sockaddr_in serv;
WSADATA wsa;
WSAStartup(MAKEWORD(2,0),&wsa);

listenfd=::WSASocket(AF_INET, SOCK_STREAM,IPPROTO_IP, NULL, 0,
WSA_FLAG_OVERLAPPED);
serv.sin_family=AF_INET;
serv.sin_port=htons(9393);
serv.sin_addr.s_addr=INADDR_ANY;
//-----------------------------------------------------------------------------------------------
printf("server port: 9393\n");
printf("client port: 8080\n\n");
len=sizeof(serv);

//server
printf("bind %d",bind(listenfd,(struct sockaddr *)&serv,len));
printf("listen %d\n",listen(listenfd,5));
printf("listen fd %d\n",listenfd);

g_main_port = ::CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL,
0,0);
g_work_port = ::CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL,
0,0);
if(g_work_port == NULL || g_main_port == NULL)
{
printf("can not create io completion port");
return ;
}
// creating 4 threads
printf("_beginthread \n");
unsigned int i;
for(i = 0; i < 4; i++)
{
unsigned int thread_id;
HANDLE thread = (HANDLE) _beginthreadex(NULL, 0, ThreadProc, 0, 0,
&thread_id);
if(thread == NULL)
{
printf("can not create thread");
return ;
}
}

HANDLE handle =::CreateIoCompletionPort((HANDLE)&listenfd,

(HANDLE)g_main_port,(int) &listenfd, 0);

DWORD* key;
DWORD read_bytes;
OVERLAPPED* over;
while(1)
{
printf("wait for GetQueuedCampletionStatus g_main_port \n");

int ret = ::GetQueuedCompletionStatus(g_main_port,
&read_bytes, (DWORD*)&key,&over, INFINITE);
// here it is waiting
printf("wait for accept ret %d read Bites %d\n",ret,read_bytes);
if(ret==0||read_bytes<=0)
{

continue ;
}

DWORD dwRead;
char *acceptbuff =(char*)malloc(1024);
OVERLAPPED ol;
bool bl;
bl=AcceptEx(listenfd,acceptfd,acceptbuff,0,sizeof( struct
sockaddr_in)+16,sizeof( struct sockaddr_in)+16,&dwRead, &ol);

printf("\naccepted............... %d bool %d \n",acceptfd,bl);
SESSION session;

session.acceptfd=acceptfd;
HANDLE hPort = ::CreateIoCompletionPort((HANDLE)&session,
(HANDLE)g_work_port, (int) &session, 0);

char *serverbuff,tbuff[1024];
int nrec=0;
serverbuff=(char*)malloc(1024);

nrec=recv(acceptfd,serverbuff,1024,0);
printf("nrecv %d \n",nrec);
strcpy(tbuff,serverbuff);

if(nrec==0 || nrec == SOCKET_ERROR )
{
printf("ERROR SERVER FD_ISSET(server[i],&rset)==0\n");
printf(" thread breaks acceptfd %d.........\n",acceptfd);
closesocket(acceptfd);
return;
}
}//End if(FD_ISSET(listenfd,&rset))
//}//End while
}
//------------------------Thread ----------------------
unsigned int WINAPI ThreadProc(void *arg)
{
printf("in thread\n");
DWORD* key;
DWORD read_bytes;
OVERLAPPED* over;

while(1)
{
printf("GetQCP g_main_port\n");
int ret = ::GetQueuedCompletionStatus(g_work_port, &read_bytes,
(DWORD*)&key,&over, 10000);
printf("ret %d read bites %d\n",ret,read_bytes);
if(ret==0||read_bytes<=0)
continue ;
bluethought
2006-05-15 19:08:33 UTC
Permalink
Strongly suggest you just go out and pick up a copy of Network
Programming for MS Windows by Jones and Ohlund. For somebody here to
explain IOCP to you is going to take very long.

In a nutshell, one simple IOCP model
[0] Main prog creates IOCP with CreateIOCP
[1] Listener thread calls accept(), adds sockets to IOCP with
CreateIOCP passing socket handle....also posts initial read/write on
socket
[2] Worker threads call GQCS, process completion notification, loop
around to GQCS

For your proxy server you'll need to do more than that....you'll need
to use connect() or ConnectEx to connect your client socket, and use
CreateIOCP after connect() or prior to ConnectEx to add your client
socket to IOCP.

Get the book! Or read the MSDN articles etc very carefully.
bluethought
2006-05-15 19:11:24 UTC
Permalink
Also check the IOCP sample code shipping with the Platform SDK or the
(painfully cryptic) code from www.mvps.org.
David Gravereaux
2006-05-25 10:43:29 UTC
Permalink
Post by sandeep
printf("listen %d\n",listen(listenfd,5));
Replace that line with 5 calls to AcceptEx() [and remount your brain
for a long relearn as IOCP is a wildly different animal].
David Gravereaux
2006-05-25 12:04:07 UTC
Permalink
On Thu, 25 May 2006 03:43:29 -0700, David Gravereaux
Post by David Gravereaux
Post by sandeep
printf("listen %d\n",listen(listenfd,5));
Replace that line with 5 calls to AcceptEx() [and remount your brain
for a long relearn as IOCP is a wildly different animal].
Sandeep,

The general concept with overlapped I/O is that you want to "preload"
calls to the socket. IOW, before anyone tries to connect to your
listening socket, call a pool of AcceptExs and already have a worker
thread looping on GQCS for the port. In this way, when someone
connects, the first AcceptEx call comes back off the port.

Notification and data are the same. Or you can think of it as the
data coming back from the port is the notification.

Here's some working code to read, if you want:
http://iocpsock.cvs.sourceforge.net/iocpsock/iocpsock/iocpsock_lolevel.c?revision=HEAD&view=markup
http://iocpsock.cvs.sourceforge.net/iocpsock/iocpsock/ws2tcp.c?revision=HEAD&view=markup
http://iocpsock.cvs.sourceforge.net/iocpsock/iocpsock

It's rather complete and fully debugged, so it might be hard to see
the general concepts. CompletionThreadProc() dequeues the port with
HandleIo() being the big meat. To make things simple for myself, I
only use one thread dequeueing the completion port. For myself, that
thread doesn't drive the rest of my application, so your decision to
use multiple threads should be based on:

1) The level of code complexity for managing many->one merging of
WSARecv buffers in mixed order may overshoot your project deadline by
a good 4 months :)

2) will the threads dequeueing your GQCS also drive your application
or is your socket code only a producer to your next stage (buffering/
protocol parser?) that might be a modeled as a consumer.

Personally, I like to chain together functional blocks in
producer/consumer so I didn't need to go to multiple completion
threads. I do have multiple consumers on my single socket producer,
but I didn't have to split my socket code down the middle and entangle
myself in the mess of WSARecv buffer merging madness ordering hell
with more than one completion worker thread ;)

All you people running multiple completion threads can call me a
cynic, but I'm so glad I didn't go down that path.

P.S. I found I could get more performance reducing contentions in the
completion routine than I could by adding more threading around it.
Even one EnterCriticalSection really slowed GQCS iteration per sec.
Loading...