AIX includes a structure for elaborate limits to be setup for process resources. These limits are defined in the file /etc/security/limits (which may be modified directly or via the command chuser).

example of output:

username:
    fsize = 2097151
    core = 2048
    cpu = 3600
    data = 131072
    rss = 65536
    stack = 8192

These settings specify absolute limits on resource usage, and they cannot be overridden by the user.

To change your user's memory use limit, you can use the following command:
 

  # chuser rss=102400 username


This command sets username's default memory use limit to 100MB by modifying or adding the rss line for their username in /etc/security/limits. As usual, the limits set in the default stanza are applied for any user without specific settings of their own. Setting a limit of -1 will allow unlimited use of that system resource.